Hi Thomas,
Ok!
Regards,
Fred
2015-03-05 4:53 GMT-03:00 Thomas Mueller <thomas.tom.muel...@gmail.com>:
> Hi,
>
> I have a new patch for the LIRS cache:
>
> Index: src/main/org/h2/mvstore/cache/CacheLongKeyLIRS.java
> ===================================================================
> --- src/main/org/h2/mvstore/cache/CacheLongKeyLIRS.java (revision 6051)
> +++ src/main/org/h2/mvstore/cache/CacheLongKeyLIRS.java (working copy)
> @@ -62,7 +62,7 @@
> *
> * @param maxMemory the maximum memory to use (1 or larger)
> */
> - public CacheLongKeyLIRS(int maxMemory) {
> + public CacheLongKeyLIRS(long maxMemory) {
> this(maxMemory, 16, 8);
> }
>
> @@ -544,11 +544,6 @@
> private final int mask;
>
> /**
> - * The LIRS stack size.
> - */
> - private int stackSize;
> -
> - /**
> * The stack of recently referenced elements. This includes all
> hot
> * entries, and the recently referenced cold entries. Resident
> cold
> * entries that were not recently referenced, as well as
> non-resident
> @@ -559,6 +554,11 @@
> private final Entry<V> stack;
>
> /**
> + * The number of entries in the stack.
> + */
> + private int stackSize;
> +
> + /**
> * The queue of resident cold entries.
> * <p>
> * There is always at least one entry: the head entry.
> @@ -800,6 +800,10 @@
> old = e.value;
> remove(key, hash);
> }
> + if (memory > maxMemory) {
> + // the new entry is too big to fit
> + return old;
> + }
> e = new Entry<V>();
> e.key = key;
> e.value = value;
> @@ -808,9 +812,15 @@
> e.mapNext = entries[index];
> entries[index] = e;
> usedMemory += memory;
> - if (usedMemory > maxMemory && mapSize > 0) {
> - // an old entry needs to be removed
> - evict(e);
> + if (usedMemory > maxMemory) {
> + // old entries needs to be removed
> + evict();
> + // if the cache is full, the new entry is
> + // cold if possible
> + if (stackSize > 0) {
> + // the new cold entry is at the top of the queue
> + addToQueue(queue, e);
> + }
> }
> mapSize++;
> // added entries are always added to the stack
> @@ -874,23 +884,22 @@
> * Evict cold entries (resident and non-resident) until the
> memory limit
> * is reached. The new entry is added as a cold entry, except if
> it is
> * the only entry.
> - *
> - * @param newCold a new cold entry
> */
> - private void evict(Entry<V> newCold) {
> + private void evict() {
> + do {
> + evictBlock();
> + } while (usedMemory > maxMemory);
> + }
> +
> + private void evictBlock() {
> // ensure there are not too many hot entries: right shift of
> 5 is
> // division by 32, that means if there are only 1/32 (3.125%)
> or
> // less cold entries, a hot entry needs to become cold
> while (queueSize <= (mapSize >>> 5) && stackSize > 0) {
> convertOldestHotToCold();
> }
> - if (stackSize > 0) {
> - // the new cold entry is at the top of the queue
> - addToQueue(queue, newCold);
> - }
> // the oldest resident cold entries become non-resident
> - // but at least one cold entry (the new one) must stay
> - while (usedMemory > maxMemory && queueSize > 1) {
> + while (usedMemory > maxMemory && queueSize > 0) {
> Entry<V> e = queue.queuePrev;
> usedMemory -= e.memory;
> removeFromQueue(e);
> Index: src/docsrc/html/changelog.html
> ===================================================================
> --- src/docsrc/html/changelog.html (revision 6061)
> +++ src/docsrc/html/changelog.html (working copy)
> @@ -17,7 +17,12 @@
> <h1>Change Log</h1>
>
> <h2>Next Version (unreleased)</h2>
> -<ul><li>-
> +<ul><li>The LIRS cache could grow larger than the allocated memory.
> +</li><li>A new file system implementation that re-opens the file if it
> was closed due
> + to the application calling Thread.interrupt(). File name prefix
> "retry:".
> + Please note it is strongly recommended to avoid calling
> Thread.interrupt;
> + this is a problem for various libraries, including Apache Lucene.
> +</li><li>MVStore: use RandomAccessFile file system if the file name
> starts with "file:".
> </li></ul>
>
> <h2>Version 1.4.186 Beta (2015-03-02)</h2>
> Index: src/test/org/h2/test/store/TestCacheLongKeyLIRS.java
> ===================================================================
> --- src/test/org/h2/test/store/TestCacheLongKeyLIRS.java (revision 6052)
> +++ src/test/org/h2/test/store/TestCacheLongKeyLIRS.java (working copy)
> @@ -43,6 +43,7 @@
> testPruneStack();
> testLimitHot();
> testLimitNonResident();
> + testLimitMemory();
> testScanResistance();
> testRandomOperations();
> }
> @@ -77,7 +78,7 @@
> private void testEdgeCases() {
> CacheLongKeyLIRS<Integer> test = createCache(1);
> test.put(1, 10, 100);
> - assertEquals(10, test.get(1).intValue());
> + assertEquals(0, test.size());
> try {
> test.put(1, null, 100);
> fail();
> @@ -335,6 +336,24 @@
> "cold: 19 non-resident: 18 17 16");
> }
>
> + private void testLimitMemory() {
> + CacheLongKeyLIRS<Integer> test = createCache(4);
> + for (int i = 0; i < 5; i++) {
> + test.put(i, 10 * i, 1);
> + }
> + verify(test, "mem: 4 stack: 4 3 2 1 cold: 4 non-resident: 0");
> + assertTrue("" + test.getUsedMemory(), test.getUsedMemory() <= 4);
> + test.put(6, 60, 3);
> + verify(test, "mem: 4 stack: 6 3 cold: 6 non-resident:");
> + assertTrue("" + test.getUsedMemory(), test.getUsedMemory() <= 4);
> + test.put(7, 70, 3);
> + verify(test, "mem: 4 stack: 7 6 3 cold: 7 non-resident: 6");
> + assertTrue("" + test.getUsedMemory(), test.getUsedMemory() <= 4);
> + test.put(8, 80, 4);
> + verify(test, "mem: 4 stack: 8 cold: non-resident:");
> + assertTrue("" + test.getUsedMemory(), test.getUsedMemory() <= 4);
> + }
> +
> private void testScanResistance() {
> boolean log = false;
> int size = 20;
>
>
> On Wed, Mar 4, 2015 at 7:30 PM, Thomas Mueller <
> thomas.tom.muel...@gmail.com> wrote:
>
>> Hi,
>>
>> Thanks a lot for the test cases and the patch! I'm still trying to
>> understand the patch, and I am trying to find a simpler way to fix the
>> problem.
>>
>> Regards,
>> Thomas
>>
>>
>> On Sunday, March 1, 2015, Fred&Dani&Pandora&Aquiles <zepf...@gmail.com>
>> wrote:
>>
>>> Hi Thomas,
>>>
>>> I attached a patch that reproduces the problem and I reviewed my
>>> previous patch that try to reduce the memory below of the limit. It passed
>>> in the regression tests too.
>>>
>>> Regards,
>>>
>>> Fred
>>>
>>>
>>>
>>> 2015-02-27 16:42 GMT-03:00 Thomas Mueller <thomas.tom.muel...@gmail.com>
>>> :
>>>
>>>> Hi,
>>>>
>>>> OK, that sounds like a bug. Do you have a simple test case for this (a
>>>> patch to TestCacheLongKeyLIRS or TestCacheLIRS?
>>>>
>>>> Regards,
>>>> Thomas
>>>>
>>>> On Fri, Feb 20, 2015 at 12:39 PM, Fred&Dani&Pandora&Aquiles <
>>>> zepf...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I expressed myself incorrectly in the previous post. The cold stack do
>>>>> not becomes empty, the size of the stack is reduced to one element, the
>>>>> newCold item. Thus, the memory used is not reduced, even with elements in
>>>>> the hot stack that could be moved to the cold stack.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Fred
>>>>>
>>>>> 2015-02-20 9:11 GMT-02:00 Fred&Dani&Pandora&Aquiles <zepf...@gmail.com
>>>>> >:
>>>>>
>>>>> Hi again,
>>>>>>
>>>>>> I don't know if would be necessary to create another thread, but
>>>>>> during my analysis of LIRS implementation I dealt with some cases where
>>>>>> the
>>>>>> cold stack becomes empty and the memory limit continued to not met. What
>>>>>> you think of the attached patch?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Fred
>>>>>>
>>>>>> 2015-02-19 16:31 GMT-02:00 Thomas Mueller <
>>>>>> thomas.tom.muel...@gmail.com>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks a lot for the test case! I can now reproduce it, and I'm
>>>>>>> working on a fix.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Thomas
>>>>>>>
>>>>>>> On Fri, Feb 13, 2015 at 2:31 AM, Trask Stalnaker <
>>>>>>> trask.stalna...@gmail.com> wrote:
>>>>>>>
>>>>>>>> The code below dumps a heap with ~86mb cacheChunkRef. If you bump
>>>>>>>> OUTER_LOOP_COUNT to 1,000,000, it dumps a heap with ~469mb
>>>>>>>> cacheChunkRef.
>>>>>>>>
>>>>>>>> public class MVStoreTest {
>>>>>>>>
>>>>>>>> private static final int OUTER_LOOP_COUNT = 100000;
>>>>>>>>
>>>>>>>> public static void main(String[] args) throws Exception {
>>>>>>>> Connection connection =
>>>>>>>>
>>>>>>>> DriverManager.getConnection("jdbc:h2:~/test;compress=true", "sa", "");
>>>>>>>>
>>>>>>>> Statement statement = connection.createStatement();
>>>>>>>> statement.execute("create table xyz (x varchar, y bigint)");
>>>>>>>> statement.execute("create index xyz_idx on xyz (x, y)");
>>>>>>>>
>>>>>>>> PreparedStatement preparedStatement =
>>>>>>>> connection.prepareStatement("insert into xyz (x, y)
>>>>>>>> values (?, ?)");
>>>>>>>>
>>>>>>>> long start = System.currentTimeMillis();
>>>>>>>> for (int i = 0; i < OUTER_LOOP_COUNT; i++) {
>>>>>>>> for (int j = 0; j < 100; j++) {
>>>>>>>> preparedStatement.setString(1, "x" + j);
>>>>>>>> preparedStatement.setLong(2, i);
>>>>>>>> preparedStatement.addBatch();
>>>>>>>> }
>>>>>>>> preparedStatement.executeBatch();
>>>>>>>> if ((i + 1) % 1000 == 0) {
>>>>>>>> long end = System.currentTimeMillis();
>>>>>>>> System.out.println((i + 1) + " " + (end - start));
>>>>>>>> start = end;
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> ManagementFactory.getPlatformMBeanServer().invoke(
>>>>>>>>
>>>>>>>> ObjectName.getInstance("com.sun.management:type=HotSpotDiagnostic"),
>>>>>>>> "dumpHeap",
>>>>>>>> new Object[] {"heapdump.hprof", true},
>>>>>>>> new String[] {"java.lang.String", "boolean"});
>>>>>>>>
>>>>>>>> connection.close();
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wednesday, February 11, 2015 at 10:56:18 PM UTC-8, Thomas
>>>>>>>> Mueller wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> That means the LIRS cache keeps too many non-resident cold
>>>>>>>>> entries. I wonder how to best reproduce this problem... Do you have a
>>>>>>>>> simple test case (a description what you do would be enough in this
>>>>>>>>> case)?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>> On Thu, Feb 12, 2015 at 3:17 AM, Trask Stalnaker <
>>>>>>>>> trask.s...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I was looking over a heap dump and was surprised by
>>>>>>>>>> MVStore.cacheChunkRef consuming 29mb memory.
>>>>>>>>>>
>>>>>>>>>> MVStore.cache is consuming 14mb which makes sense given the 16mb
>>>>>>>>>> default cache limit.
>>>>>>>>>>
>>>>>>>>>> Eclipse MemoryAnalyzer OQL shows 100,000+
>>>>>>>>>> org.h2.mvstore.cache.CacheLongKeyLIRS$Entry
>>>>>>>>>> objects with memory = 0 and value = null:
>>>>>>>>>>
>>>>>>>>>> select * from org.glowroot.shaded.h2.mvstore.cache.
>>>>>>>>>> CacheLongKeyLIRS$Entry
>>>>>>>>>> where value = null and memory = 0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is this memory consumption expected? Is there a concern memory
>>>>>>>>>> may grow unbounded given enough of these objects with memory = 0
>>>>>>>>>> since H2
>>>>>>>>>> won't count them against it's internal memory limit?
>>>>>>>>>>
>>>>>>>>>> Using latest 1.4.185.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Trask
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "H2 Database" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to h2-database...@googlegroups.com.
>>>>>>>>>> To post to this group, send email to h2-da...@googlegroups.com.
>>>>>>>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "H2 Database" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to h2-database+unsubscr...@googlegroups.com.
>>>>>>>> To post to this group, send email to h2-database@googlegroups.com.
>>>>>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "H2 Database" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to h2-database+unsubscr...@googlegroups.com.
>>>>>>> To post to this group, send email to h2-database@googlegroups.com.
>>>>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "H2 Database" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to h2-database+unsubscr...@googlegroups.com.
>>>>> To post to this group, send email to h2-database@googlegroups.com.
>>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "H2 Database" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to h2-database+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to h2-database@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "H2 Database" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to h2-database+unsubscr...@googlegroups.com.
>>> To post to this group, send email to h2-database@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/h2-database.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "H2 Database" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to h2-database+unsubscr...@googlegroups.com.
> To post to this group, send email to h2-database@googlegroups.com.
> Visit this group at http://groups.google.com/group/h2-database.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups "H2
Database" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to h2-database+unsubscr...@googlegroups.com.
To post to this group, send email to h2-database@googlegroups.com.
Visit this group at http://groups.google.com/group/h2-database.
For more options, visit https://groups.google.com/d/optout.
