[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106165#comment-14106165 ]
Nick Dimiduk commented on HBASE-11331: -------------------------------------- Thanks for having a look [~stack]. bq. You figure the compressor issue, our not reusing them? I've punted on the compressor is a non-issue, but I haven't run with a profiler yet. I think it was related to the non-native gz impl while running locally. I'll re-enable tracing there as well with the next runs and see what I see. bq. This is best case (when =false, we are seeking? Or is it always inside fscache?) Yes, this configuration is targeting a best case for this patch. The fscache is minimized with this config, seems to stay down around 3.5g (vs 11.5g blockcache). Compression ratio is reported as 0.2437, so --size=45 should be ~11g compressed -- larger than the fscache. Because the PE test is random, I believe we'll be thrashing the fscache with both =true and =false. The iowait charts indicate both configs are doing io constantly, just more with =false (as expected). bq. What is the 'cost' keeping stuff compressed? What if you do a run where all fits in cache, for both cases? I'm testing a couple more scenarios, this one was already on the list. > [blockcache] lazy block decompression > ------------------------------------- > > Key: HBASE-11331 > URL: https://issues.apache.org/jira/browse/HBASE-11331 > Project: HBase > Issue Type: Improvement > Components: regionserver > Reporter: Nick Dimiduk > Assignee: Nick Dimiduk > Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, > HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch, > HBASE-11331.05.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, > lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, > lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true-16h.pdf, > v03-20g-045g-true.pdf > > > Maintaining data in its compressed form in the block cache will greatly > increase our effective blockcache size and should show a meaning improvement > in cache hit rates in well designed applications. The idea here is to lazily > decompress/decrypt blocks when they're consumed, rather than as soon as > they're pulled off of disk. > This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)