Hi Thomas,
I attached a patch that reproduces the problem and I reviewed my previous
patch that try to reduce the memory below of the limit. It passed in the
regression tests too.
Regards,
Fred
2015-02-27 16:42 GMT-03:00 Thomas Mueller <thomas.tom.muel...@gmail.com>:
> Hi,
>
> OK, that sounds like a bug. Do you have a simple test case for this (a
> patch to TestCacheLongKeyLIRS or TestCacheLIRS?
>
> Regards,
> Thomas
>
> On Fri, Feb 20, 2015 at 12:39 PM, Fred&Dani&Pandora&Aquiles <
> zepf...@gmail.com> wrote:
>
>> Hi,
>>
>> I expressed myself incorrectly in the previous post. The cold stack do
>> not becomes empty, the size of the stack is reduced to one element, the
>> newCold item. Thus, the memory used is not reduced, even with elements in
>> the hot stack that could be moved to the cold stack.
>>
>> Regards,
>>
>> Fred
>>
>> 2015-02-20 9:11 GMT-02:00 Fred&Dani&Pandora&Aquiles <zepf...@gmail.com>:
>>
>> Hi again,
>>>
>>> I don't know if would be necessary to create another thread, but during
>>> my analysis of LIRS implementation I dealt with some cases where the cold
>>> stack becomes empty and the memory limit continued to not met. What you
>>> think of the attached patch?
>>>
>>> Regards,
>>>
>>> Fred
>>>
>>> 2015-02-19 16:31 GMT-02:00 Thomas Mueller <thomas.tom.muel...@gmail.com>
>>> :
>>>
>>>> Hi,
>>>>
>>>> Thanks a lot for the test case! I can now reproduce it, and I'm working
>>>> on a fix.
>>>>
>>>> Regards,
>>>> Thomas
>>>>
>>>> On Fri, Feb 13, 2015 at 2:31 AM, Trask Stalnaker <
>>>> trask.stalna...@gmail.com> wrote:
>>>>
>>>>> The code below dumps a heap with ~86mb cacheChunkRef. If you bump
>>>>> OUTER_LOOP_COUNT to 1,000,000, it dumps a heap with ~469mb cacheChunkRef.
>>>>>
>>>>> public class MVStoreTest {
>>>>>
>>>>> private static final int OUTER_LOOP_COUNT = 100000;
>>>>>
>>>>> public static void main(String[] args) throws Exception {
>>>>> Connection connection =
>>>>>
>>>>> DriverManager.getConnection("jdbc:h2:~/test;compress=true", "sa", "");
>>>>>
>>>>> Statement statement = connection.createStatement();
>>>>> statement.execute("create table xyz (x varchar, y bigint)");
>>>>> statement.execute("create index xyz_idx on xyz (x, y)");
>>>>>
>>>>> PreparedStatement preparedStatement =
>>>>> connection.prepareStatement("insert into xyz (x, y)
>>>>> values (?, ?)");
>>>>>
>>>>> long start = System.currentTimeMillis();
>>>>> for (int i = 0; i < OUTER_LOOP_COUNT; i++) {
>>>>> for (int j = 0; j < 100; j++) {
>>>>> preparedStatement.setString(1, "x" + j);
>>>>> preparedStatement.setLong(2, i);
>>>>> preparedStatement.addBatch();
>>>>> }
>>>>> preparedStatement.executeBatch();
>>>>> if ((i + 1) % 1000 == 0) {
>>>>> long end = System.currentTimeMillis();
>>>>> System.out.println((i + 1) + " " + (end - start));
>>>>> start = end;
>>>>> }
>>>>> }
>>>>>
>>>>> ManagementFactory.getPlatformMBeanServer().invoke(
>>>>>
>>>>> ObjectName.getInstance("com.sun.management:type=HotSpotDiagnostic"),
>>>>> "dumpHeap",
>>>>> new Object[] {"heapdump.hprof", true},
>>>>> new String[] {"java.lang.String", "boolean"});
>>>>>
>>>>> connection.close();
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> On Wednesday, February 11, 2015 at 10:56:18 PM UTC-8, Thomas Mueller
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> That means the LIRS cache keeps too many non-resident cold entries.
>>>>>> I wonder how to best reproduce this problem... Do you have a simple test
>>>>>> case (a description what you do would be enough in this case)?
>>>>>>
>>>>>> Regards,
>>>>>> Thomas
>>>>>>
>>>>>> On Thu, Feb 12, 2015 at 3:17 AM, Trask Stalnaker <
>>>>>> trask.s...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I was looking over a heap dump and was surprised by
>>>>>>> MVStore.cacheChunkRef consuming 29mb memory.
>>>>>>>
>>>>>>> MVStore.cache is consuming 14mb which makes sense given the 16mb
>>>>>>> default cache limit.
>>>>>>>
>>>>>>> Eclipse MemoryAnalyzer OQL shows 100,000+
>>>>>>> org.h2.mvstore.cache.CacheLongKeyLIRS$Entry
>>>>>>> objects with memory = 0 and value = null:
>>>>>>>
>>>>>>> select * from org.glowroot.shaded.h2.mvstore.cache.
>>>>>>> CacheLongKeyLIRS$Entry
>>>>>>> where value = null and memory = 0
>>>>>>>
>>>>>>>
>>>>>>> Is this memory consumption expected? Is there a concern memory may
>>>>>>> grow unbounded given enough of these objects with memory = 0 since H2
>>>>>>> won't
>>>>>>> count them against it's internal memory limit?
>>>>>>>
>>>>>>> Using latest 1.4.185.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Trask
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "H2 Database" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to h2-database...@googlegroups.com.
>>>>>>> To post to this group, send email to h2-da...@googlegroups.com.
>>>>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "H2 Database" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to h2-database+unsubscr...@googlegroups.com.
>>>>> To post to this group, send email to h2-database@googlegroups.com.
>>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "H2 Database" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to h2-database+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to h2-database@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/h2-database.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "H2 Database" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to h2-database+unsubscr...@googlegroups.com.
>> To post to this group, send email to h2-database@googlegroups.com.
>> Visit this group at http://groups.google.com/group/h2-database.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "H2 Database" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to h2-database+unsubscr...@googlegroups.com.
> To post to this group, send email to h2-database@googlegroups.com.
> Visit this group at http://groups.google.com/group/h2-database.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups "H2
Database" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to h2-database+unsubscr...@googlegroups.com.
To post to this group, send email to h2-database@googlegroups.com.
Visit this group at http://groups.google.com/group/h2-database.
For more options, visit https://groups.google.com/d/optout.
# This patch file was generated by NetBeans IDE
# Following Index: paths are relative to: /home/fred/Dropbox/h2-trunk/trunk/h2/src/test/org/h2/test/store
# This patch can be applied using context Tools: Patch action on respective folder.
# It uses platform neutral UTF-8 encoding and \n newlines.
# Above lines and this line are ignored by the patching process.
Index: TestCacheLongKeyLIRS.java
--- TestCacheLongKeyLIRS.java Base (BASE)
+++ TestCacheLongKeyLIRS.java Modificado Localmente (Baseado em LOCAL)
@@ -43,6 +43,7 @@
testPruneStack();
testLimitHot();
testLimitNonResident();
+ testLimitMemory();
testScanResistance();
testRandomOperations();
}
@@ -335,6 +336,20 @@
"cold: 19 non-resident: 18 17 16");
}
+ private void testLimitMemory() {
+ CacheLongKeyLIRS<Integer> test = createCache(4);
+ for (int i = 0; i < 5; i++) {
+ test.put(i, 10 * i, 1);
+ }
+ assertTrue(test.getUsedMemory()<=4l);
+ test.put(6, 6* 11, 3);
+ assertTrue(test.getUsedMemory()<=4l);
+ test.put(7, 7* 12, 3);
+ assertTrue(test.getUsedMemory()<=4l);
+ test.put(8, 8* 13, 4);
+ assertTrue(test.getUsedMemory()<=4l);
+ }
+
private void testScanResistance() {
boolean log = false;
int size = 20;
# This patch file was generated by NetBeans IDE
# Following Index: paths are relative to: /home/fred/Dropbox/h2-trunk/trunk/h2/src/main/org/h2/mvstore/cache
# This patch can be applied using context Tools: Patch action on respective folder.
# It uses platform neutral UTF-8 encoding and \n newlines.
# Above lines and this line are ignored by the patching process.
Index: CacheLongKeyLIRS.java
--- CacheLongKeyLIRS.java Base (BASE)
+++ CacheLongKeyLIRS.java Modificado Localmente (Baseado em LOCAL)
@@ -62,7 +62,7 @@
*
* @param maxMemory the maximum memory to use (1 or larger)
*/
- public CacheLongKeyLIRS(int maxMemory) {
+ public CacheLongKeyLIRS(long maxMemory) {
this(maxMemory, 16, 8);
}
@@ -890,10 +890,18 @@
}
// the oldest resident cold entries become non-resident
// but at least one cold entry (the new one) must stay
- while (usedMemory > maxMemory && queueSize > 1) {
+ while (usedMemory > maxMemory && queueSize >= 1) {
Entry<V> e = queue.queuePrev;
- usedMemory -= e.memory;
removeFromQueue(e);
+ //Do not change the newCold
+ if (e == newCold){
+ //Check the memory limit and continues the memory release process
+ if (usedMemory > maxMemory && queueSize < 1 && stackSize > 0) {
+ convertOldestHotToCold();
+ }
+ continue;
+ }
+ usedMemory -= e.memory;
e.value = null;
e.memory = 0;
addToQueue(queue2, e);
@@ -903,8 +911,19 @@
int hash = getHash(e.key);
remove(e.key, hash);
}
+ /*
+ * There's no more cold elements and the used memory is still higher than the limit. So, move the hot entries to the cold stack until the memory limit be reached.
+ */
+ if (usedMemory > maxMemory && queueSize < 1 && stackSize > 0) {
+ convertOldestHotToCold();
}
}
+ //Checks if the new cold was removed from the cold stack and add it again
+ if (newCold.isHot() && stackSize > 0){
+ //The newCold was removed previously
+ addToQueue(queue, newCold);
+ }
+ }
private void convertOldestHotToCold() {
// the last entry of the stack is known to be hot
