[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: 11678v3.098.txt Commit to 0.98 needed an addendum. Here is what I applied. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.098.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 0.98+. Cherry-picked. Fixed javadoc warning on commit. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: 11678v3.txt Why ain't this running? Retry. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: 11678v3.txt Fix some findbugs in blockcache area > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.txt, gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: 11678v2.txt Thanks [~ndimiduk] Will wait see what hadoopqa says. I think there a findbugs or two in here. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: gc_crash_unevenblocks_with_lots_of_evictions.png Here is how it used to run crashing after 30 minutes when violent evictions were running. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: 11678v2.txt Managed to get a test in there for each of the modes that can happen adding item to BucketCache. Lets see what the findbugs issues are. Test failures look unrelated. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11678: --- Fix Version/s: 0.98.6 2.0.0 0.99.0 > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: gc_over_12_hours_unevenblocks_with_lots_of_evictions.png Ran for 12 hours. GC plot looks clean. It may be tending up slightly... will run for longer next time but looking at heap dump, this ramCache issue is no longer the culprit. Looking to make a unit test. Its a bit awkward given how deep down doDrain is inside BucketCache but worth doing I'd say. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11678: --- Priority: Critical (was: Major) Raising priority though. Seems hot read / high load is likely where this cache would be employed. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch This seems to be working. When we failed add an entry failing with a CacheFullException -- a common occurance when heavy read load and lots of cache misses -- we would drop the entry were were trying to add failing to remove it from the onheap Map. Over time the onheap Map would fill with these orphans. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Status: Patch Available (was: Open) > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.98.5, 0.99.0, 2.0.0 >Reporter: stack >Assignee: stack > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Component/s: BlockCache Affects Version/s: 2.0.0 0.99.0 0.98.5 > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
