[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101748#comment-17101748
]
Josh Elser commented on HBASE-23887:
------------------------------------
Can you please update the title/description so your improvement is a little
more clear? I had to dig into your patch to actually understand what you were
doing to get this performance improvement on your workload.
As I understand it: Given a scenario where you have more data than can fit into
block cache and you have a high rate of evictions, you choose to not cache a
block which we would have otherwise chosen to cache under the assumption that
this "churn": to cache the N+1th block, we have to evict the Nth block.
One big assumption I see with this is that your data is being uniformly
accessed. At least, your solution would have a higher benefit if all data is
accessed "equally". Or maybe re-stated, your improvement only works if the data
being accessed frequently (the middle band of your zipfian distribution) does
not fit into block cache.
What YCSB workload did you run for which these results you've shared. Also, how
much data did you generate and how does that relate to the total blockcache
size?
> BlockCache performance improve
> ------------------------------
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
> Reporter: Danil Lipovoy
> Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png, cmp.png,
> evict_BC100_vs_BC23.png, read_requests_100pBC_vs_23pBC.png
>
>
> НГHi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC. See the picture in attachment with test below. Requests per second is
> higher, GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 0-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
