[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135184#comment-17135184
]
Danil Lipovoy commented on HBASE-23887:
---------------------------------------
And the log the second try (the feature is enabled):
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0,
current caching DataBlock (%): 100
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0,
current caching DataBlock (%): 100
evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0,
current caching DataBlock (%): 100
evicted (MB): -905, ratio 0.0, overhead (%): -552, heavy eviction counter: 0,
current caching DataBlock (%): 100 < start the test
evicted (MB): 4676, ratio 1.07, overhead (%): 2238, heavy eviction counter: 1,
current caching DataBlock (%): 85
evicted (MB): 4561, ratio 1.05, overhead (%): 2180, heavy eviction counter: 2,
current caching DataBlock (%): 70
evicted (MB): 3535, ratio 1.04, overhead (%): 1667, heavy eviction counter: 3,
current caching DataBlock (%): 55
evicted (MB): 2508, ratio 1.02, overhead (%): 1154, heavy eviction counter: 4,
current caching DataBlock (%): 44
evicted (MB): 1824, ratio 0.88, overhead (%): 812, heavy eviction counter: 5,
current caching DataBlock (%): 36
evicted (MB): 1255, ratio 0.61, overhead (%): 527, heavy eviction counter: 6,
current caching DataBlock (%): 31
evicted (MB): 912, ratio 0.41, overhead (%): 356, heavy eviction counter: 7,
current caching DataBlock (%): 28
evicted (MB): 684, ratio 0.32, overhead (%): 242, heavy eviction counter: 8,
current caching DataBlock (%): 26
evicted (MB): 570, ratio 0.29, overhead (%): 185, heavy eviction counter: 9,
current caching DataBlock (%): 25
evicted (MB): 342, ratio 0.24, overhead (%): 71, heavy eviction counter: 10,
current caching DataBlock (%): 25
evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 11,
current caching DataBlock (%): 25
evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 12,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 13,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 14,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.08, overhead (%): 14, heavy eviction counter: 15,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.07, overhead (%): 14, heavy eviction counter: 16,
current caching DataBlock (%): 25
evicted (MB): 223, ratio 0.06, overhead (%): 11, heavy eviction counter: 17,
current caching DataBlock (%): 25
evicted (MB): 107, ratio 0.06, overhead (%): -47, heavy eviction counter: 17,
current caching DataBlock (%): 30 < back pressure
evicted (MB): 456, ratio 0.16, overhead (%): 128, heavy eviction counter: 18,
current caching DataBlock (%): 29
evicted (MB): 456, ratio 0.19, overhead (%): 128, heavy eviction counter: 19,
current caching DataBlock (%): 28
evicted (MB): 456, ratio 0.2, overhead (%): 128, heavy eviction counter: 20,
current caching DataBlock (%): 27
evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 21,
current caching DataBlock (%): 27
evicted (MB): 342, ratio 0.17, overhead (%): 71, heavy eviction counter: 22,
current caching DataBlock (%): 27
evicted (MB): 342, ratio 0.16, overhead (%): 71, heavy eviction counter: 23,
current caching DataBlock (%): 27
evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 24,
current caching DataBlock (%): 27
evicted (MB): 342, ratio 0.13, overhead (%): 71, heavy eviction counter: 25,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 26,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 27,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 28,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 29,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 30,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 31,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 32,
current caching DataBlock (%): 27
evicted (MB): 228, ratio 0.09, overhead (%): 14, heavy eviction counter: 33,
current caching DataBlock (%): 27
evicted (MB): 1026, ratio 0.42, overhead (%): 413, heavy eviction counter: 34,
current caching DataBlock (%): 23 < added gets
evicted (MB): 1140, ratio 0.66, overhead (%): 470, heavy eviction counter: 35,
current caching DataBlock (%): 19
evicted (MB): 913, ratio 0.75, overhead (%): 356, heavy eviction counter: 36,
current caching DataBlock (%): 16
evicted (MB): 798, ratio 0.72, overhead (%): 299, heavy eviction counter: 37,
current caching DataBlock (%): 14
evicted (MB): 684, ratio 0.62, overhead (%): 242, heavy eviction counter: 38,
current caching DataBlock (%): 12
evicted (MB): 456, ratio 0.51, overhead (%): 128, heavy eviction counter: 39,
current caching DataBlock (%): 11
evicted (MB): 342, ratio 0.42, overhead (%): 71, heavy eviction counter: 40,
current caching DataBlock (%): 11
evicted (MB): 342, ratio 0.35, overhead (%): 71, heavy eviction counter: 41,
current caching DataBlock (%): 11
evicted (MB): 342, ratio 0.32, overhead (%): 71, heavy eviction counter: 42,
current caching DataBlock (%): 11
evicted (MB): 342, ratio 0.32, overhead (%): 71, heavy eviction counter: 43,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.3, overhead (%): 14, heavy eviction counter: 44,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.29, overhead (%): 14, heavy eviction counter: 45,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.27, overhead (%): 14, heavy eviction counter: 46,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.25, overhead (%): 14, heavy eviction counter: 47,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.22, overhead (%): 14, heavy eviction counter: 48,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.21, overhead (%): 14, heavy eviction counter: 49,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.19, overhead (%): 14, heavy eviction counter: 50,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.18, overhead (%): 14, heavy eviction counter: 51,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.17, overhead (%): 14, heavy eviction counter: 52,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.17, overhead (%): 14, heavy eviction counter: 53,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.16, overhead (%): 14, heavy eviction counter: 54,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.15, overhead (%): 14, heavy eviction counter: 55,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.15, overhead (%): 14, heavy eviction counter: 56,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.15, overhead (%): 14, heavy eviction counter: 57,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.14, overhead (%): 14, heavy eviction counter: 58,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.14, overhead (%): 14, heavy eviction counter: 59,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.14, overhead (%): 14, heavy eviction counter: 60,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.14, overhead (%): 14, heavy eviction counter: 61,
current caching DataBlock (%): 11
evicted (MB): 228, ratio 0.14, overhead (%): 14, heavy eviction counter: 62,
current caching DataBlock (%): 11
evicted (MB): 226, ratio 0.13, overhead (%): 13, heavy eviction counter: 63,
current caching DataBlock (%): 11
evicted (MB): 82, ratio 0.12, overhead (%): -59, heavy eviction counter: 63,
current caching DataBlock (%): 17 < interrupt gets
evicted (MB): 456, ratio 0.22, overhead (%): 128, heavy eviction counter: 64,
current caching DataBlock (%): 16
evicted (MB): 221, ratio 0.16, overhead (%): 10, heavy eviction counter: 65,
current caching DataBlock (%): 16
evicted (MB): 53, ratio 0.12, overhead (%): -74, heavy eviction counter: 65,
current caching DataBlock (%): 24 < back pressure
evicted (MB): 456, ratio 0.23, overhead (%): 128, heavy eviction counter: 66,
current caching DataBlock (%): 23
evicted (MB): 228, ratio 0.18, overhead (%): 14, heavy eviction counter: 67,
current caching DataBlock (%): 23
evicted (MB): 66, ratio 0.14, overhead (%): -67, heavy eviction counter: 67,
current caching DataBlock (%): 30 < back pressure
evicted (MB): 570, ratio 0.28, overhead (%): 185, heavy eviction counter: 68,
current caching DataBlock (%): 29
evicted (MB): 456, ratio 0.33, overhead (%): 128, heavy eviction counter: 69,
current caching DataBlock (%): 28
evicted (MB): 456, ratio 0.31, overhead (%): 128, heavy eviction counter: 70,
current caching DataBlock (%): 27
evicted (MB): 456, ratio 0.3, overhead (%): 128, heavy eviction counter: 71,
current caching DataBlock (%): 26
evicted (MB): 456, ratio 0.27, overhead (%): 128, heavy eviction counter: 72,
current caching DataBlock (%): 25
evicted (MB): 342, ratio 0.24, overhead (%): 71, heavy eviction counter: 73,
current caching DataBlock (%): 25
evicted (MB): 342, ratio 0.21, overhead (%): 71, heavy eviction counter: 74,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.19, overhead (%): 14, heavy eviction counter: 75,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.18, overhead (%): 14, heavy eviction counter: 76,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.16, overhead (%): 14, heavy eviction counter: 77,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.15, overhead (%): 14, heavy eviction counter: 78,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.14, overhead (%): 14, heavy eviction counter: 79,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.14, overhead (%): 14, heavy eviction counter: 80,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.13, overhead (%): 14, heavy eviction counter: 81,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 82,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 83,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 84,
current caching DataBlock (%): 25
evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 85,
current caching DataBlock (%): 25
evicted (MB): 225, ratio 0.11, overhead (%): 12, heavy eviction counter: 86,
current caching DataBlock (%): 25
evicted (MB): 112, ratio 0.11, overhead (%): -44, heavy eviction counter: 86,
current caching DataBlock (%): 30 < back pressure
evicted (MB): 570, ratio 0.25, overhead (%): 185, heavy eviction counter: 87,
current caching DataBlock (%): 29
evicted (MB): 456, ratio 0.3, overhead (%): 128, heavy eviction counter: 88,
current caching DataBlock (%): 28
evicted (MB): 456, ratio 0.31, overhead (%): 128, heavy eviction counter: 89,
current caching DataBlock (%): 27
evicted (MB): 456, ratio 0.3, overhead (%): 128, heavy eviction counter: 90,
current caching DataBlock (%): 26
evicted (MB): 342, ratio 0.28, overhead (%): 71, heavy eviction counter: 91,
current caching DataBlock (%): 26
evicted (MB): 342, ratio 0.26, overhead (%): 71, heavy eviction counter: 92,
current caching DataBlock (%): 26
evicted (MB): 342, ratio 0.23, overhead (%): 71, heavy eviction counter: 93,
current caching DataBlock (%): 26
evicted (MB): 342, ratio 0.21, overhead (%): 71, heavy eviction counter: 94,
current caching DataBlock (%): 26
evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 95,
current caching DataBlock (%): 26
evicted (MB): 228, ratio 0.18, overhead (%): 14, heavy eviction counter: 96,
current caching DataBlock (%): 26
evicted (MB): 228, ratio 0.18, overhead (%): 14, heavy eviction counter: 97,
current caching DataBlock (%): 26
evicted (MB): 201, ratio 0.18, overhead (%): 0, heavy eviction counter: 98,
current caching DataBlock (%): 26
evicted (MB): 0, ratio 0.0, overhead (%): -100, heavy eviction counter: 0,
current caching DataBlock (%): 100
Why the ratio much less - as I understand there are two reason:
1. We keep a lot of META blocks and skip data blocks. We use meta quite often,
so it populate the cache.
2. We are skipping the same blocks (for example where the last digits 50-100)
and keep blocks which used again and again. Low eviction rate lets them to be
reused.
So, when we add a lot of get came the new blocks and the ratio is growing.
When the back pressure is coming some of data blocks populate the single cache
and the ratio grows again. Looks like it doesn't matter what kind of reading
we have (scan or gets). I meant the situation when the table is scanning again
and again. In opposite (one time scan) much better just disable the caching.
> BlockCache performance improve by reduce eviction rate
> ------------------------------------------------------
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
> Reporter: Danil Lipovoy
> Assignee: Danil Lipovoy
> Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
