[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200618#comment-17200618
]
Danil Lipovoy commented on HBASE-23887:
---------------------------------------
I am going to write some article about this and have done some test CS vs HB.
Think it could be interesting.
I run YCSB from 2 hosts (800 threads summary) on tables which size:
HBase - 300 GB on HDFS (100 GB pure data)
Cassandra - 250 GB (replication factor = 3)
It means the volume approximately the same (HB a little bit more).
The HB parameters:
_dfs.client.short.circuit.num = 5 - this is another my improvement
https://issues.apache.org/jira/browse/HDFS-15202 - it helps to speed up HB more_
_hbase.lru.cache.heavy.eviction.count.limit = 30 - it means the patch will work
after 30 evictions (~5 minutes)_
_hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - good target for eviction_
So, I aggregated logs YCSB and put this into Excel:
!image-2020-09-23-10-06-11-189.png!
At the beginning CS faster then HB. When _heavy.eviction.count.limit_ pass 30
was enables the improvement and performance become the same.
How it looks into the log of RegionServer:
_2020-09-22 18:31:47,561 INFO org...LruBlockCache: BlockCache evicted (MB):
7238, overhead (%): 2312, heavy eviction counter: 21, current caching DataBlock
(%): 100_
_2020-09-22 18:31:57,808 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 22, current caching DataBlock
(%): 100_
_2020-09-22 18:32:08,051 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 23, current caching DataBlock
(%): 100_
_2020-09-22 18:32:18,155 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 24, current caching DataBlock
(%): 100_
_2020-09-22 18:32:28,479 INFO org...LruBlockCache: BlockCache evicted (MB):
7238, overhead (%): 2312, heavy eviction counter: 25, current caching DataBlock
(%): 100_
_2020-09-22 18:32:38,754 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 26, current caching DataBlock
(%): 100_
_2020-09-22 18:32:49,334 INFO org...LruBlockCache: BlockCache evicted (MB):
7238, overhead (%): 2312, heavy eviction counter: 27, current caching DataBlock
(%): 100_
_2020-09-22 18:32:59,712 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 28, current caching DataBlock
(%): 100_
_2020-09-22 18:33:10,061 INFO org...LruBlockCache: BlockCache evicted (MB):
7238, overhead (%): 2312, heavy eviction counter: 29, current caching DataBlock
(%): 100_
_2020-09-22 18:33:20,220 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 30, current caching DataBlock
(%): 100 <- the feature enabled_
_2020-09-22 18:33:30,314 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 31, current caching DataBlock
(%): 85_
_2020-09-22 18:33:41,390 INFO org...LruBlockCache: BlockCache evicted (MB):
6721, overhead (%): 2140, heavy eviction counter: 32, current caching DataBlock
(%): 70_
_2020-09-22 18:33:52,281 INFO org...LruBlockCache: BlockCache evicted (MB):
5687, overhead (%): 1795, heavy eviction counter: 33, current caching DataBlock
(%): 55_
_2020-09-22 18:34:03,394 INFO org...LruBlockCache: BlockCache evicted (MB):
4136, overhead (%): 1278, heavy eviction counter: 34, current caching DataBlock
(%): 43_
_2020-09-22 18:34:15,088 INFO org...LruBlockCache: BlockCache evicted (MB):
2585, overhead (%): 761, heavy eviction counter: 35, current caching DataBlock
(%): 36_
_2020-09-22 18:34:27,752 INFO org...LruBlockCache: BlockCache evicted (MB):
1551, overhead (%): 417, heavy eviction counter: 36, current caching DataBlock
(%): 32_
_2020-09-22 18:34:45,233 INFO org...LruBlockCache: BlockCache evicted (MB):
940, overhead (%): 213, heavy eviction counter: 37, current caching DataBlock
(%): 30_
_2020-09-22 18:34:55,364 INFO org...LruBlockCache: BlockCache evicted (MB):
289, overhead (%): -4, heavy eviction counter: 37, current caching DataBlock
(%): 31_
_2020-09-22 18:35:05,466 INFO org...LruBlockCache: BlockCache evicted (MB):
240, overhead (%): -20, heavy eviction counter: 37, current caching DataBlock
(%): 34_
_2020-09-22 18:35:15,564 INFO org...LruBlockCache: BlockCache evicted (MB):
254, overhead (%): -16, heavy eviction counter: 37, current caching DataBlock
(%): 36_
_2020-09-22 18:35:25,670 INFO org...LruBlockCache: BlockCache evicted (MB):
279, overhead (%): -7, heavy eviction counter: 37, current caching DataBlock
(%): 37_
_2020-09-22 18:35:35,801 INFO org...LruBlockCache: BlockCache evicted (MB):
294, overhead (%): -2, heavy eviction counter: 37, current caching DataBlock
(%): 38_
_2020-09-22 18:35:45,918 INFO org...LruBlockCache: BlockCache evicted (MB):
309, overhead (%): 3, heavy eviction counter: 38, current caching DataBlock
(%): 38_
_2020-09-22 18:35:56,027 INFO org...LruBlockCache: BlockCache evicted (MB):
253, overhead (%): -16, heavy eviction counter: 38, current caching DataBlock
(%): 40_
_2020-09-22 18:36:06,164 INFO org...LruBlockCache: BlockCache evicted (MB):
368, overhead (%): 22, heavy eviction counter: 39, current caching DataBlock
(%): 40_
_2020-09-22 18:36:16,283 INFO org...LruBlockCache: BlockCache evicted (MB):
302, overhead (%): 0, heavy eviction counter: 39, current caching DataBlock
(%): 41_
_2020-09-22 18:36:26,403 INFO org...LruBlockCache: BlockCache evicted (MB):
346, overhead (%): 15, heavy eviction counter: 40, current caching DataBlock
(%): 41_
_2020-09-22 18:36:36,517 INFO org...LruBlockCache: BlockCache evicted (MB):
304, overhead (%): 1, heavy eviction counter: 41, current caching DataBlock
(%): 41_
_2020-09-22 18:36:46,626 INFO org...LruBlockCache: BlockCache evicted (MB):
263, overhead (%): -13, heavy eviction counter: 41, current caching DataBlock
(%): 43_
_2020-09-22 18:36:56,757 INFO org...LruBlockCache: BlockCache evicted (MB):
408, overhead (%): 36, heavy eviction counter: 42, current caching DataBlock
(%): 43_
_2020-09-22 18:37:06,880 INFO org...LruBlockCache: BlockCache evicted (MB):
373, overhead (%): 24, heavy eviction counter: 43, current caching DataBlock
(%): 43_
_2020-09-22 18:37:16,993 INFO org...LruBlockCache: BlockCache evicted (MB):
323, overhead (%): 7, heavy eviction counter: 44, current caching DataBlock
(%): 43_
_2020-09-22 18:37:27,137 INFO org...LruBlockCache: BlockCache evicted (MB):
291, overhead (%): -3, heavy eviction counter: 44, current caching DataBlock
(%): 44_
_2020-09-22 18:37:37,271 INFO org...LruBlockCache: BlockCache evicted (MB):
391, overhead (%): 30, heavy eviction counter: 45, current caching DataBlock
(%): 44_
_2020-09-22 18:37:47,386 INFO org...LruBlockCache: BlockCache evicted (MB):
357, overhead (%): 19, heavy eviction counter: 46, current caching DataBlock
(%): 44_
_2020-09-22 18:37:57,510 INFO org...LruBlockCache: BlockCache evicted (MB):
322, overhead (%): 7, heavy eviction counter: 47, current caching DataBlock
(%): 44_
_2020-09-22 18:38:07,647 INFO org...LruBlockCache: BlockCache evicted (MB):
301, overhead (%): 0, heavy eviction counter: 47, current caching DataBlock
(%): 45_
_2020-09-22 18:38:17,775 INFO org...LruBlockCache: BlockCache evicted (MB):
396, overhead (%): 32, heavy eviction counter: 48, current caching DataBlock
(%): 45_
_2020-09-22 18:38:27,906 INFO org...LruBlockCache: BlockCache evicted (MB):
371, overhead (%): 23, heavy eviction counter: 49, current caching DataBlock
(%): 45_
_2020-09-22 18:38:38,028 INFO org...LruBlockCache: BlockCache evicted (MB):
344, overhead (%): 14, heavy eviction counter: 50, current caching DataBlock
(%): 45_
_2020-09-22 18:38:48,170 INFO org...LruBlockCache: BlockCache evicted (MB):
322, overhead (%): 7, heavy eviction counter: 51, current caching DataBlock
(%): 45_
_2020-09-22 18:38:58,287 INFO org...LruBlockCache: BlockCache evicted (MB):
294, overhead (%): -2, heavy eviction counter: 51, current caching DataBlock
(%): 46_
_2020-09-22 18:39:08,411 INFO org...LruBlockCache: BlockCache evicted (MB):
413, overhead (%): 37, heavy eviction counter: 52, current caching DataBlock
(%): 46_
_2020-09-22 18:39:18,554 INFO org...LruBlockCache: BlockCache evicted (MB):
384, overhead (%): 28, heavy eviction counter: 53, current caching DataBlock
(%): 46_
_2020-09-22 18:39:28,673 INFO org...LruBlockCache: BlockCache evicted (MB):
357, overhead (%): 19, heavy eviction counter: 54, current caching DataBlock
(%): 46_
_2020-09-22 18:39:38,836 INFO org...LruBlockCache: BlockCache evicted (MB):
347, overhead (%): 15, heavy eviction counter: 55, current caching DataBlock
(%): 46_
_2020-09-22 18:39:48,962 INFO org...LruBlockCache: BlockCache evicted (MB):
323, overhead (%): 7, heavy eviction counter: 56, current caching DataBlock
(%): 46_
_2020-09-22 18:39:59,104 INFO org...LruBlockCache: BlockCache evicted (MB):
308, overhead (%): 2, heavy eviction counter: 57, current caching DataBlock
(%): 46_
_2020-09-22 18:40:09,223 INFO org...LruBlockCache: BlockCache evicted (MB):
289, overhead (%): -4, heavy eviction counter: 57, current caching DataBlock
(%): 47_
_2020-09-22 18:40:19,382 INFO org...LruBlockCache: BlockCache evicted (MB):
413, overhead (%): 37, heavy eviction counter: 58, current caching DataBlock
(%): 47_
_2020-09-22 18:40:29,555 INFO org...LruBlockCache: BlockCache evicted (MB):
389, overhead (%): 29, heavy eviction counter: 59, current caching DataBlock
(%): 47_
_2020-09-22 18:40:39,658 INFO org...LruBlockCache: BlockCache evicted (MB):
271, overhead (%): -10, heavy eviction counter: 59, current caching DataBlock
(%): 49_
_2020-09-22 18:40:49,658 INFO org...LruBlockCache: BlockCache evicted (MB): 0,
overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%):
100_
_2020-09-22 18:40:59,729 INFO org...LruBlockCache: BlockCache evicted (MB): 2,
overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%):
100_
_2020-09-22 18:41:09,729 INFO org...LruBlockCache: BlockCache evicted (MB): 0,
overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%):
100_
All YCSB logs in the attachment - [^ycsb_logs.zip]
> BlockCache performance improve by reduce eviction rate
> ------------------------------------------------------
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
> Reporter: Danil Lipovoy
> Assignee: Danil Lipovoy
> Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
