[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: *Operations per second (batch 1000)* | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108|< not good| |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average of 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal hbase.regionserver.throughput.controlle
[jira] [Commented] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293952#comment-17293952 ] Danil Lipovoy commented on HBASE-25619: --- Just one - home PC > 50% reading performance degradation 2.4.1 over 1.6.0 > > > Key: HBASE-25619 > URL: https://issues.apache.org/jira/browse/HBASE-25619 > Project: HBase > Issue Type: Bug >Reporter: Danil Lipovoy >Priority: Major > Attachments: logs.zip, scripts.zip > > > I have found performance issues. YCSB tests show: > *Operations per second (batch 1000)* > | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| > |INSERTS|68|68|75|76|< this is fine| > |GETS|92|100|72|48|< 50% less than 1.6.0| > |FLUSHED GETS|126|141|120|108|< not good| > |GET+INSERT|69|71|68|66| | > > GETS - means gets right after inserts. > FLUSHED GETS - after flush and major compation > All numbers are average of 3 runs. > For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: > — run 01 hdl300_LRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 > — run 02 hdl300_LRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 > — run 03 hdl300_LRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 > But always were 4 runs (not 3). First run for warm up and excluded from > aggregation (usually it is faster then all runs later). > All test done with AdaptiveLRU > (https://issues.apache.org/jira/browse/HBASE-23887) > This is because: > # RS on old LRU just often fall under pressure. > # It is faster than current version (much faster when server powerful). > For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, > SSD) this is current version LRU (1.4.13): > -- > --- run 01 hdl300_oldLRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 > --- run 02 hdl300_oldLRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 > --- run 03 hdl300_oldLRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 > This is new version (1.4.13): > – run 01 hdl300_newLRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 > — run 02 hdl300_newLRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 > — run 03 hdl300_newLRU_thr30_reg100 — > thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 > thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 > All test done with the same params: > > > hbase.cluster.distributed > true > > > hbase.tmp.dir > ./tmp/hb > > > hbase.rootdir > /tmp/hbase > > > hbase.unsafe.stream.capability.enforce > false > > > zookeeper.session.timeout > 12 > > > hbase.rpc.timeout > 12 > > > hbase.regionserver.handler.count > 300 > > > hbase.regionserver.metahandler.count > 30 > > > hbase.regionserver.maxlogs > 200 > > > hbase.hregion.memstore.flush.size > 1342177280 > > > hbase.hregion.memstore.block.multiplier > 6 > > > hbase.hstore.compactionThreshold > 2 > > > hbase.hstore.blockingStoreFiles > 200 > >
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: *Operations per second (batch 1000)* | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108|< not good| |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average of 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: *Operations per second (batch 1000)* | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less| |FLUSHED GETS|126|141|120|108|< not good | |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average of 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22G ZK is
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: *Operations per second (batch 1000)* | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less| |FLUSHED GETS|126|141|120|108|< not good| |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average of 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22G ZK is se
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: | |*Operations per second (batch 1000)*| | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*| |INSERTS|68|68|75|76| |GETS|92|100|72|48| |FLUSHED GETS|126|141|120|108| |GET+INSERT|69|71|68|66| GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average of 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22G ZK is separate (downloaded from apache site) because
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: | | | |*Operations per second (batch 1000)*| | | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108| | |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average of 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=2
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: | |*Operations per second (batch 1000)*| | | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108| | |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average of 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got from: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22G ZK
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: | |*Operations per second (batch 1000)*| | | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108| | |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average over 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got form: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor, 32 GB MEM, SSD) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22G
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: | |*Operations per second (batch 1000)*| | | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108| | |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average over 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got form: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22G ZK is separate (
[jira] [Updated] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-25619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25619: -- Description: I have found performance issues. YCSB tests show: | |*Operations per second (batch 1000)*| | | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108| | |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average over 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got form: — run 01 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 02 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 — run 03 hdl300_LRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor) this is current version LRU (1.4.13): -- --- run 01 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): – run 01 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 — run 02 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 — run 03 hdl300_newLRU_thr30_reg100 — thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 hbase.lru.cache.heavy.eviction.overhead.coefficient 0.01 hbase.wal.provider multiwal And everywhere export HBASE_HEAPSIZE=22G ZK is separate (
[jira] [Created] (HBASE-25619) 50% reading performance degradation 2.4.1 over 1.6.0
Danil Lipovoy created HBASE-25619: - Summary: 50% reading performance degradation 2.4.1 over 1.6.0 Key: HBASE-25619 URL: https://issues.apache.org/jira/browse/HBASE-25619 Project: HBase Issue Type: Bug Reporter: Danil Lipovoy Attachments: logs.zip, scripts.zip I have found performance issues. YCSB tests show: | |*Operations per second (batch 1000)*| | | |*1.4.13*|*1.6.0*|*2.2.6*|*2.4.1*|*comments*| |INSERTS|68|68|75|76|< this is fine| |GETS|92|100|72|48|< 50% less than 1.6.0| |FLUSHED GETS|126|141|120|108| | |GET+INSERT|69|71|68|66| | GETS - means gets right after inserts. FLUSHED GETS - after flush and major compation All numbers are average over 3 runs. For example GETS 2.4.1 => (45 + 49 + 50) / 3 = 48 got form: --- run 01 hdl300_LRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 45 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 --- run 02 hdl300_LRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 109 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 49 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 77 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 --- run 03 hdl300_LRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 108 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 50 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 But always were 4 runs (not 3). First run for warm up and excluded from aggregation (usually it is faster then all runs later). All test done with AdaptiveLRU (https://issues.apache.org/jira/browse/HBASE-23887) This is because: # RS on old LRU just often fall under pressure. # It is faster than current version (much faster when server powerful). For example on my PC (AMD Ryzen 7 2700X Eight-Core Processor) this is current version LRU (1.4.13): --- run 01 hdl300_oldLRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 76 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 65 --- run 02 hdl300_oldLRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 115 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 81 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 --- run 03 hdl300_oldLRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 116 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 82 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 66 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 66 This is new version (1.4.13): -- run 01 hdl300_newLRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 128 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 67 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 70 --- run 02 hdl300_newLRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 126 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 93 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 69 --- run 03 hdl300_newLRU_thr30_reg100 --- thr30 cnt10 tim300 num0 max1 bch1000 reg100 fget ops= 125 thr30 cnt10 tim300 num0 max1 bch1000 reg100 get ops= 91 thr30 cnt10 tim300 num0 max1 bch1000 reg100 ins ops= 68 thr30 cnt10 tim300 num0 max1 bch1000 reg100 upd ops= 67 All test done with the same params: hbase.cluster.distributed true hbase.tmp.dir ./tmp/hb hbase.rootdir /tmp/hbase hbase.unsafe.stream.capability.enforce false zookeeper.session.timeout 12 hbase.rpc.timeout 12 hbase.regionserver.handler.count 300 hbase.regionserver.metahandler.count 30 hbase.regionserver.maxlogs 200 hbase.hregion.memstore.flush.size 1342177280 hbase.hregion.memstore.block.multiplier 6 hbase.hstore.compactionThreshold 2 hbase.hstore.blockingStoreFiles 200 hbase.regionserver.optionalcacheflushinterval 1800 hbase.regionserver.thread.compaction.large 12 hbase.regionserver.wal.enablecompression true hbase.server.compactchecker.interval.multiplier 200 hbase.rest.threads.min 8 hbase.rest.threads.max 150 hbase.thrift.minWorkerThreads 200 hbase.regionserver.thread.compaction.small 6 hbase.ipc.server.read.threadpool.size 60 hbase.lru.cache.heavy.eviction.count.limit 0 hbase.lru.cache.heavy.eviction.mb.size.limit 200 h
[jira] [Resolved] (HBASE-25123) Add possibility to set different types of L1 cache
[ https://issues.apache.org/jira/browse/HBASE-25123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy resolved HBASE-25123. --- Release Note: Was implemented Resolution: Implemented > Add possibility to set different types of L1 cache > -- > > Key: HBASE-25123 > URL: https://issues.apache.org/jira/browse/HBASE-25123 > Project: HBase > Issue Type: New Feature > Components: BlockCache >Reporter: Danil Lipovoy >Priority: Minor > > The feature HBASE-23887 allow speed up to 3 times read performance but maybe > it is too complicated. So there are proposals give users possibility to > choose type of cache L1. Looks like it needs to change few classes > (CombinedBlockCache, InclusiveCombinedBlockCache, CacheConfig) if somebody > can code this it would be cool. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23887) New L1 cache : AdaptiveLRU
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284373#comment-17284373
]
Danil Lipovoy commented on HBASE-23887:
---
[~vjasani] thanks for your active interest!
[~busbey] could you please explain where to write the release note?
And I think it would be good give explanation into: = Apache HBase (TM)
Reference Guide
Hope is it ok?
> New L1 cache : AdaptiveLRU
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2
>
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, PR#1257.diff,
> cmp.png, evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282015#comment-17282015
]
Danil Lipovoy commented on HBASE-23887:
---
[~vjasani] Thanks for reasonable comments! Fixed
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, PR#1257.diff,
> cmp.png, evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281245#comment-17281245
]
Danil Lipovoy edited comment on HBASE-23887 at 2/8/21, 5:34 PM:
[~vjasani] could you please take a look at the PR
[https://github.com/apache/hbase/pull/2934] ?
was (Author: pustota):
[~vjasani] could you please take a look at the PR
[https://github.com/apache/hbase/pull/2934] ?
There were some problems with rebase of previous PR so I made the new.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, PR#1257.diff,
> cmp.png, evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to ma
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281245#comment-17281245
]
Danil Lipovoy commented on HBASE-23887:
---
[~vjasani] could you please take a look at the PR
[https://github.com/apache/hbase/pull/2934] ?
There were some problems with rebase of previous PR so I made the new.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, PR#1257.diff,
> cmp.png, evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261183#comment-17261183
]
Danil Lipovoy commented on HBASE-23887:
---
[~vjasani], thanks a lot for explaining!
I am going to see how to make 1-6 soon and will write here when ready create
the PR.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210050#comment-17210050
]
Danil Lipovoy edited comment on HBASE-23887 at 10/8/20, 8:04 AM:
-
Looks like nobody wants to develop HBASE-25123 (I think it would make the code
much more complicated), so easier and simple just merge this PR. All
complicated math in one function EvictionThread->run() and well documented.
was (Author: pustota):
Looks like nobody wants to develop HBASE-25123 (I think it would make the code
much more complicated), so easier and simple just merge this PR. All
complicated math just in one function EvictionThread->run() and well
documented.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 t
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210050#comment-17210050
]
Danil Lipovoy edited comment on HBASE-23887 at 10/8/20, 8:04 AM:
-
Looks like nobody wants to develop HBASE-25123 (I think it would make the code
much more complicated), so easier and simple just merge this PR. All
complicated math just in one function EvictionThread->run() and well
documented.
was (Author: pustota):
Looks like nobody wants to develop HBASE-25123 (I think it would make the code
much more complicated), so easier and simple just merge this PR. All
complicated math just in one function evic() and well documented.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 6
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210050#comment-17210050
]
Danil Lipovoy commented on HBASE-23887:
---
Looks like nobody wants to develop HBASE-25123 (I think it would make the code
much more complicated), so easier and simple just merge this PR. All
complicated math just in one function evic() and well documented.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian J
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 10:52 AM:
--
Thank you for your interest)
I created HBASE-25123 and maybe somebody will release possibility to set
classes L1 realisation.
was (Author: pustota):
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and nothing happen. We are just discussing what would be good to do
(more performance tests, resolve conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case although is saving huge amount of CPU but something
went wrong and I have no idea why. Maybe because HBASE-24915 has priority
"major" but I set just "minor";))
So I created HBASE-25123 and maybe somebody will release possibility to set
classes L1 realisation.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
>
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 10:49 AM:
--
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and nothing happen. We are just discussing what would be good to do
(more performance tests, resolve conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case although is saving huge amount of CPU but something
went wrong and I have no idea why. Maybe because HBASE-24915 has priority
"major" but I set just "minor";))
So I created HBASE-25123 and maybe somebody will release possibility to set
classes L1 realisation.
was (Author: pustota):
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen. We
are just discussing what would be good to do (more performance tests, resolve
conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case although is saving huge amount of CPU but something
went wrong and I have no idea why. Maybe because HBASE-24915 has priority
"major" but I set just "minor";))
So I created HBASE-25123 and maybe somebody will release possibility to set
classes L1 realisation.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlo
[jira] [Updated] (HBASE-25123) Add possibility to set different types of L1 cache
[ https://issues.apache.org/jira/browse/HBASE-25123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25123: -- Description: The feature HBASE-23887 allow speed up to 3 times read performance but maybe it is too complicated. So there are proposals give users possibility to choose type of cache L1. Looks like it needs to change few classes (CombinedBlockCache, InclusiveCombinedBlockCache, CacheConfig) if somebody can code this it would be cool. (was: The feature HBASE-23887 allow speed up to 3 times read performance but maybe it is too complicated. So there are proposals give users possibility to choose type of cache L1. Looks like it needs to change few classes (CombinedBlockCache, InclusiveCombinedBlockCache, CacheConfig) ) > Add possibility to set different types of L1 cache > -- > > Key: HBASE-25123 > URL: https://issues.apache.org/jira/browse/HBASE-25123 > Project: HBase > Issue Type: New Feature >Reporter: Danil Lipovoy >Priority: Minor > > The feature HBASE-23887 allow speed up to 3 times read performance but maybe > it is too complicated. So there are proposals give users possibility to > choose type of cache L1. Looks like it needs to change few classes > (CombinedBlockCache, InclusiveCombinedBlockCache, CacheConfig) if somebody > can code this it would be cool. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25123) Add possibility to set different types of L1 cache
[ https://issues.apache.org/jira/browse/HBASE-25123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danil Lipovoy updated HBASE-25123: -- Description: The feature HBASE-23887 allow speed up to 3 times read performance but maybe it is too complicated. So there are proposals give users possibility to choose type of cache L1. Looks like it needs to change few classes (CombinedBlockCache, InclusiveCombinedBlockCache, CacheConfig) (was: The feature HBASE-23887 allow speed up to 3 times read performance but maybe it is too complicated. So there are proposals give users possibility to choose type of cache L1. ) > Add possibility to set different types of L1 cache > -- > > Key: HBASE-25123 > URL: https://issues.apache.org/jira/browse/HBASE-25123 > Project: HBase > Issue Type: New Feature >Reporter: Danil Lipovoy >Priority: Minor > > The feature HBASE-23887 allow speed up to 3 times read performance but maybe > it is too complicated. So there are proposals give users possibility to > choose type of cache L1. Looks like it needs to change few classes > (CombinedBlockCache, InclusiveCombinedBlockCache, CacheConfig) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523 ] Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 10:44 AM: -- Thank you for your interest) But it looks like the feature will never merged because it has dragged on for 7 month and more then dozen developers watching on this and nothing happen. We are just discussing what would be good to do (more performance tests, resolve conflicts etc) and I can't see the end of that. I don't understand how open source development works, sometimes some created issue like HBASE-24915 which save 1% CPU and it is apply without discussion in 1 day. But in this case although is saving huge amount of CPU but something went wrong and I have no idea why. Maybe because HBASE-24915 has priority "major" but I set just "minor";)) So I created HBASE-25123 and maybe somebody will release possibility to set classes L1 realisation. was (Author: pustota): Thank you for your interest) But it looks like the feature will never merged because it has dragged on for 7 month and more then dozen developers watching on this and nothing happen. We are just discussing what would be good to do (more performance tests, resolve conflicts etc) and I can't see the end of that. I don't understand how open source development works, sometimes some created issue like HBASE-24915 which save 1% CPU and it is apply without discussion in 1 day. But in this case although is saving huge amount of CPU but something went wrong and I have no idea why. Maybe because HBASE-24915 has priority "major" but I set just "minor";)) So I created HBASE-25123 and maybe somebody released possibility to set classes L1 realisation. > BlockCache performance improve by reduce eviction rate > -- > > Key: HBASE-23887 > URL: https://issues.apache.org/jira/browse/HBASE-23887 > Project: HBase > Issue Type: Improvement > Components: BlockCache, Performance >Reporter: Danil Lipovoy >Assignee: Danil Lipovoy >Priority: Minor > Attachments: 1582787018434_rs_metrics.jpg, > 1582801838065_rs_metrics_new.png, BC_LongRun.png, > BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png, > evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png, > eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png, > image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png, > image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png, > image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png, > image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png, > image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png, > ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png, > requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png, > scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip > > > Hi! > I first time here, correct me please if something wrong. > All latest information is here: > [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing] > I want propose how to improve performance when data in HFiles much more than > BlockChache (usual story in BigData). The idea - caching only part of DATA > blocks. It is good becouse LruBlockCache starts to work and save huge amount > of GC. > Sometimes we have more data than can fit into BlockCache and it is cause a > high rate of evictions. In this case we can skip cache a block N and insted > cache the N+1th block. Anyway we would evict N block quite soon and that why > that skipping good for performance. > --- > Some information below isn't actual > --- > > > Example: > Imagine we have little cache, just can fit only 1 block and we are trying to > read 3 blocks with offsets: > 124 > 198 > 223 > Current way - we put the block 124, then put 198, evict 124, put 223, evict > 198. A lot of work (5 actions). > With the feature - last few digits evenly distributed from 0 to 99. When we > divide by modulus we got: > 124 -> 24 > 198 -> 98 > 223 -> 23 > It helps to sort them. Some part, for example below 50 (if we set > *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip > others. It means we will not try to handle the block 198 and save CPU for > other job. In the result - we put block 124, then put 223, evict 124 (3 > actions). > See the picture in attachment with test below. Requests per second is higher, > GC is lower. > > The key point of the code: > Added the parameter: *hbase.lru.cache.data.block.percent* which by default = > 100 > > But if we set it 1-99, then will work the next logic: >
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 10:43 AM:
--
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen. We
are just discussing what would be good to do (more performance tests, resolve
conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case although is saving huge amount of CPU but something
went wrong and I have no idea why. Maybe because HBASE-24915 has priority
"major" but I set just "minor";))
So I created HBASE-25123 and maybe somebody released possibility to set
classes L1 realisation.
was (Author: pustota):
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen. We
are just discussing what would be good me to do (more performance tests,
resolve conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case although is saving huge amount of CPU but something
went wrong and I have no idea why. Maybe because HBASE-24915 has priority
"major" but I set just "minor";))
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
>
[jira] [Created] (HBASE-25123) Add possibility to set different types of L1 cache
Danil Lipovoy created HBASE-25123: - Summary: Add possibility to set different types of L1 cache Key: HBASE-25123 URL: https://issues.apache.org/jira/browse/HBASE-25123 Project: HBase Issue Type: New Feature Reporter: Danil Lipovoy The feature HBASE-23887 allow speed up to 3 times read performance but maybe it is too complicated. So there are proposals give users possibility to choose type of cache L1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 10:29 AM:
--
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen. We
just discussing what would be good me to do (more performance tests, resolve
conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case saving huge amount of CPU but something went wrong and
I have no idea why. Maybe because HBASE-24915 has priority "major" but I set
just "minor";))
was (Author: pustota):
Thank you for interesting)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case something going wrong and I have no idea why. Maybe
because HBASE-24915 has priority "major" but I set just "minor";))
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabl
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 10:31 AM:
--
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen. We
are just discussing what would be good me to do (more performance tests,
resolve conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case although is saving huge amount of CPU but something
went wrong and I have no idea why. Maybe because HBASE-24915 has priority
"major" but I set just "minor";))
was (Author: pustota):
Thank you for your interest)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen. We
just discussing what would be good me to do (more performance tests, resolve
conflicts etc) and I can't see the end of that.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case saving huge amount of CPU but something went wrong and
I have no idea why. Maybe because HBASE-24915 has priority "major" but I set
just "minor";))
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.ge
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 7:48 AM:
-
Thank you for interesting)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case something going wrong and I have no idea why. Maybe
because HBASE-24915 has priority "major" but I set just "minor";))
was (Author: pustota):
Thank you for interesting)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case something going wrong and I have no idea why.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy edited comment on HBASE-23887 at 9/30/20, 7:41 AM:
-
Thank you for interesting)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen.
I don't understand how open source development works, sometimes some created
issue like HBASE-24915 which save 1% CPU and it is apply without discussion in
1 day. But in this case something going wrong and I have no idea why.
was (Author: pustota):
Thank you for interesting)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen.
I don't understand how open source development works, sometimes some created
issue like [HBASE-24915|https://issues.apache.org/jira/browse/HBASE-24915]
which save 1% CPU and it is apply without discussion in 3 days. But in this
case something going wrong and I have no idea why.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204523#comment-17204523
]
Danil Lipovoy commented on HBASE-23887:
---
Thank you for interesting)
But it looks like the feature will never merged because it has dragged on for 7
month and more then dozen developers watching on this and nothing happen.
I don't understand how open source development works, sometimes some created
issue like [HBASE-24915|https://issues.apache.org/jira/browse/HBASE-24915]
which save 1% CPU and it is apply without discussion in 3 days. But in this
case something going wrong and I have no idea why.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (onl
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201524#comment-17201524
]
Danil Lipovoy commented on HBASE-23887:
---
Is there a simple way to specify the new class? Looks like it leads to change
few classes (CombinedBlockCache, InclusiveCombinedBlockCache, CacheConfig) and
maybe will become even more complicated?
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#80300
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200618#comment-17200618 ] Danil Lipovoy edited comment on HBASE-23887 at 9/23/20, 4:04 PM: - I am going to write some article about this and have done some test Cassandra vs HBase. Think it could be interesting. I run YCSB from 2 hosts (800 threads summary) on tables which size: HBase - 300 GB on HDFS (100 GB pure data) Cassandra - 250 GB (replication factor = 3) It means the volume approximately the same (HB a little bit more). The HB parameters: _dfs.client.short.circuit.num = 5 - this is another my improvement https://issues.apache.org/jira/browse/HDFS-15202 - it helps to speed up HB more_ _hbase.lru.cache.heavy.eviction.count.limit = 30 - it means the patch will work after 30 evictions (~5 minutes)_ _hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - good target for eviction_ So, I aggregated logs YCSB and put this into Excel: !image-2020-09-23-10-06-11-189.png! At the beginning CS faster then HB. When _heavy.eviction.count.limit_ pass 30, then the feature was enabled and performance become the same. How it looks into the log of RegionServer: _2020-09-22 18:31:47,561 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 21, current caching DataBlock (%): 100_ _2020-09-22 18:31:57,808 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 22, current caching DataBlock (%): 100_ _2020-09-22 18:32:08,051 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 23, current caching DataBlock (%): 100_ _2020-09-22 18:32:18,155 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 24, current caching DataBlock (%): 100_ _2020-09-22 18:32:28,479 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 25, current caching DataBlock (%): 100_ _2020-09-22 18:32:38,754 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 26, current caching DataBlock (%): 100_ _2020-09-22 18:32:49,334 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 27, current caching DataBlock (%): 100_ _2020-09-22 18:32:59,712 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 28, current caching DataBlock (%): 100_ _2020-09-22 18:33:10,061 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 29, current caching DataBlock (%): 100_ _2020-09-22 18:33:20,220 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 30, current caching DataBlock (%): 100 <- the feature enabled here because reached hbase.lru.cache.heavy.eviction.count.limit = 30_ _2020-09-22 18:33:30,314 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 31, current caching DataBlock (%): 85_ _2020-09-22 18:33:41,390 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 32, current caching DataBlock (%): 70_ _2020-09-22 18:33:52,281 INFO org...: BlockCache evicted (MB): 5687, overhead (%): 1795, heavy eviction counter: 33, current caching DataBlock (%): 55_ _2020-09-22 18:34:03,394 INFO org...: BlockCache evicted (MB): 4136, overhead (%): 1278, heavy eviction counter: 34, current caching DataBlock (%): 43_ _2020-09-22 18:34:15,088 INFO org...: BlockCache evicted (MB): 2585, overhead (%): 761, heavy eviction counter: 35, current caching DataBlock (%): 36_ _2020-09-22 18:34:27,752 INFO org...: BlockCache evicted (MB): 1551, overhead (%): 417, heavy eviction counter: 36, current caching DataBlock (%): 32_ _2020-09-22 18:34:45,233 INFO org...: BlockCache evicted (MB): 940, overhead (%): 213, heavy eviction counter: 37, current caching DataBlock (%): 30_ _2020-09-22 18:34:55,364 INFO org...: BlockCache evicted (MB): 289, overhead (%): -4, heavy eviction counter: 37, current caching DataBlock (%): 31_ _2020-09-22 18:35:05,466 INFO org...: BlockCache evicted (MB): 240, overhead (%): -20, heavy eviction counter: 37, current caching DataBlock (%): 34_ _2020-09-22 18:35:15,564 INFO org...: BlockCache evicted (MB): 254, overhead (%): -16, heavy eviction counter: 37, current caching DataBlock (%): 36_ _2020-09-22 18:35:25,670 INFO org...: BlockCache evicted (MB): 279, overhead (%): -7, heavy eviction counter: 37, current caching DataBlock (%): 37_ _2020-09-22 18:35:35,801 INFO org...: BlockCache evicted (MB): 294, overhead (%): -2, heavy eviction counter: 37, current caching DataBlock (%): 38_ _2020-09-22 18:35:45,918 INFO org...: BlockCache evicted (MB): 309, overhead (%): 3, heavy eviction counter: 38, current caching DataBlock (%): 38_ _2020-09-22 18:35:56,027 INFO org...: BlockCache evicted (MB): 253
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200618#comment-17200618 ] Danil Lipovoy edited comment on HBASE-23887 at 9/23/20, 7:16 AM: - I am going to write some article about this and have done some test Cassandra vs HBase. Think it could be interesting. I run YCSB from 2 hosts (800 threads summary) on tables which size: HBase - 300 GB on HDFS (100 GB pure data) Cassandra - 250 GB (replication factor = 3) It means the volume approximately the same (HB a little bit more). The HB parameters: _dfs.client.short.circuit.num = 5 - this is another my improvement https://issues.apache.org/jira/browse/HDFS-15202 - it helps to speed up HB more_ _hbase.lru.cache.heavy.eviction.count.limit = 30 - it means the patch will work after 30 evictions (~5 minutes)_ _hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - good target for eviction_ So, I aggregated logs YCSB and put this into Excel: !image-2020-09-23-10-06-11-189.png! At the beginning CS faster then HB. When _heavy.eviction.count.limit_ pass 30 was enables the improvement and performance become the same. How it looks into the log of RegionServer: _2020-09-22 18:31:47,561 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 21, current caching DataBlock (%): 100_ _2020-09-22 18:31:57,808 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 22, current caching DataBlock (%): 100_ _2020-09-22 18:32:08,051 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 23, current caching DataBlock (%): 100_ _2020-09-22 18:32:18,155 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 24, current caching DataBlock (%): 100_ _2020-09-22 18:32:28,479 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 25, current caching DataBlock (%): 100_ _2020-09-22 18:32:38,754 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 26, current caching DataBlock (%): 100_ _2020-09-22 18:32:49,334 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 27, current caching DataBlock (%): 100_ _2020-09-22 18:32:59,712 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 28, current caching DataBlock (%): 100_ _2020-09-22 18:33:10,061 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 29, current caching DataBlock (%): 100_ _2020-09-22 18:33:20,220 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 30, current caching DataBlock (%): 100 <- the feature enabled here because reached hbase.lru.cache.heavy.eviction.count.limit = 30_ _2020-09-22 18:33:30,314 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 31, current caching DataBlock (%): 85_ _2020-09-22 18:33:41,390 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 32, current caching DataBlock (%): 70_ _2020-09-22 18:33:52,281 INFO org...: BlockCache evicted (MB): 5687, overhead (%): 1795, heavy eviction counter: 33, current caching DataBlock (%): 55_ _2020-09-22 18:34:03,394 INFO org...: BlockCache evicted (MB): 4136, overhead (%): 1278, heavy eviction counter: 34, current caching DataBlock (%): 43_ _2020-09-22 18:34:15,088 INFO org...: BlockCache evicted (MB): 2585, overhead (%): 761, heavy eviction counter: 35, current caching DataBlock (%): 36_ _2020-09-22 18:34:27,752 INFO org...: BlockCache evicted (MB): 1551, overhead (%): 417, heavy eviction counter: 36, current caching DataBlock (%): 32_ _2020-09-22 18:34:45,233 INFO org...: BlockCache evicted (MB): 940, overhead (%): 213, heavy eviction counter: 37, current caching DataBlock (%): 30_ _2020-09-22 18:34:55,364 INFO org...: BlockCache evicted (MB): 289, overhead (%): -4, heavy eviction counter: 37, current caching DataBlock (%): 31_ _2020-09-22 18:35:05,466 INFO org...: BlockCache evicted (MB): 240, overhead (%): -20, heavy eviction counter: 37, current caching DataBlock (%): 34_ _2020-09-22 18:35:15,564 INFO org...: BlockCache evicted (MB): 254, overhead (%): -16, heavy eviction counter: 37, current caching DataBlock (%): 36_ _2020-09-22 18:35:25,670 INFO org...: BlockCache evicted (MB): 279, overhead (%): -7, heavy eviction counter: 37, current caching DataBlock (%): 37_ _2020-09-22 18:35:35,801 INFO org...: BlockCache evicted (MB): 294, overhead (%): -2, heavy eviction counter: 37, current caching DataBlock (%): 38_ _2020-09-22 18:35:45,918 INFO org...: BlockCache evicted (MB): 309, overhead (%): 3, heavy eviction counter: 38, current caching DataBlock (%): 38_ _2020-09-22 18:35:56,027 INFO org...: BlockCache evicted (MB): 253,
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200618#comment-17200618 ] Danil Lipovoy edited comment on HBASE-23887 at 9/23/20, 7:15 AM: - I am going to write some article about this and have done some test CS vs HB. Think it could be interesting. I run YCSB from 2 hosts (800 threads summary) on tables which size: HBase - 300 GB on HDFS (100 GB pure data) Cassandra - 250 GB (replication factor = 3) It means the volume approximately the same (HB a little bit more). The HB parameters: _dfs.client.short.circuit.num = 5 - this is another my improvement https://issues.apache.org/jira/browse/HDFS-15202 - it helps to speed up HB more_ _hbase.lru.cache.heavy.eviction.count.limit = 30 - it means the patch will work after 30 evictions (~5 minutes)_ _hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - good target for eviction_ So, I aggregated logs YCSB and put this into Excel: !image-2020-09-23-10-06-11-189.png! At the beginning CS faster then HB. When _heavy.eviction.count.limit_ pass 30 was enables the improvement and performance become the same. How it looks into the log of RegionServer: _2020-09-22 18:31:47,561 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 21, current caching DataBlock (%): 100_ _2020-09-22 18:31:57,808 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 22, current caching DataBlock (%): 100_ _2020-09-22 18:32:08,051 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 23, current caching DataBlock (%): 100_ _2020-09-22 18:32:18,155 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 24, current caching DataBlock (%): 100_ _2020-09-22 18:32:28,479 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 25, current caching DataBlock (%): 100_ _2020-09-22 18:32:38,754 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 26, current caching DataBlock (%): 100_ _2020-09-22 18:32:49,334 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 27, current caching DataBlock (%): 100_ _2020-09-22 18:32:59,712 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 28, current caching DataBlock (%): 100_ _2020-09-22 18:33:10,061 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 29, current caching DataBlock (%): 100_ _2020-09-22 18:33:20,220 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 30, current caching DataBlock (%): 100 <- the feature enabled here because reached hbase.lru.cache.heavy.eviction.count.limit = 30_ _2020-09-22 18:33:30,314 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 31, current caching DataBlock (%): 85_ _2020-09-22 18:33:41,390 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 32, current caching DataBlock (%): 70_ _2020-09-22 18:33:52,281 INFO org...: BlockCache evicted (MB): 5687, overhead (%): 1795, heavy eviction counter: 33, current caching DataBlock (%): 55_ _2020-09-22 18:34:03,394 INFO org...: BlockCache evicted (MB): 4136, overhead (%): 1278, heavy eviction counter: 34, current caching DataBlock (%): 43_ _2020-09-22 18:34:15,088 INFO org...: BlockCache evicted (MB): 2585, overhead (%): 761, heavy eviction counter: 35, current caching DataBlock (%): 36_ _2020-09-22 18:34:27,752 INFO org...: BlockCache evicted (MB): 1551, overhead (%): 417, heavy eviction counter: 36, current caching DataBlock (%): 32_ _2020-09-22 18:34:45,233 INFO org...: BlockCache evicted (MB): 940, overhead (%): 213, heavy eviction counter: 37, current caching DataBlock (%): 30_ _2020-09-22 18:34:55,364 INFO org...: BlockCache evicted (MB): 289, overhead (%): -4, heavy eviction counter: 37, current caching DataBlock (%): 31_ _2020-09-22 18:35:05,466 INFO org...: BlockCache evicted (MB): 240, overhead (%): -20, heavy eviction counter: 37, current caching DataBlock (%): 34_ _2020-09-22 18:35:15,564 INFO org...: BlockCache evicted (MB): 254, overhead (%): -16, heavy eviction counter: 37, current caching DataBlock (%): 36_ _2020-09-22 18:35:25,670 INFO org...: BlockCache evicted (MB): 279, overhead (%): -7, heavy eviction counter: 37, current caching DataBlock (%): 37_ _2020-09-22 18:35:35,801 INFO org...: BlockCache evicted (MB): 294, overhead (%): -2, heavy eviction counter: 37, current caching DataBlock (%): 38_ _2020-09-22 18:35:45,918 INFO org...: BlockCache evicted (MB): 309, overhead (%): 3, heavy eviction counter: 38, current caching DataBlock (%): 38_ _2020-09-22 18:35:56,027 INFO org...: BlockCache evicted (MB): 253, overhead (%
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200618#comment-17200618 ] Danil Lipovoy edited comment on HBASE-23887 at 9/23/20, 7:13 AM: - I am going to write some article about this and have done some test CS vs HB. Think it could be interesting. I run YCSB from 2 hosts (800 threads summary) on tables which size: HBase - 300 GB on HDFS (100 GB pure data) Cassandra - 250 GB (replication factor = 3) It means the volume approximately the same (HB a little bit more). The HB parameters: _dfs.client.short.circuit.num = 5 - this is another my improvement https://issues.apache.org/jira/browse/HDFS-15202 - it helps to speed up HB more_ _hbase.lru.cache.heavy.eviction.count.limit = 30 - it means the patch will work after 30 evictions (~5 minutes)_ _hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - good target for eviction_ So, I aggregated logs YCSB and put this into Excel: !image-2020-09-23-10-06-11-189.png! At the beginning CS faster then HB. When _heavy.eviction.count.limit_ pass 30 was enables the improvement and performance become the same. How it looks into the log of RegionServer: _2020-09-22 18:31:47,561 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 21, current caching DataBlock (%): 100_ _2020-09-22 18:31:57,808 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 22, current caching DataBlock (%): 100_ _2020-09-22 18:32:08,051 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 23, current caching DataBlock (%): 100_ _2020-09-22 18:32:18,155 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 24, current caching DataBlock (%): 100_ _2020-09-22 18:32:28,479 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 25, current caching DataBlock (%): 100_ _2020-09-22 18:32:38,754 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 26, current caching DataBlock (%): 100_ _2020-09-22 18:32:49,334 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 27, current caching DataBlock (%): 100_ _2020-09-22 18:32:59,712 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 28, current caching DataBlock (%): 100_ _2020-09-22 18:33:10,061 INFO org...: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 29, current caching DataBlock (%): 100_ _2020-09-22 18:33:20,220 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 30, current caching DataBlock (%): 100 <- the feature enabled_ _2020-09-22 18:33:30,314 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 31, current caching DataBlock (%): 85_ _2020-09-22 18:33:41,390 INFO org...: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 32, current caching DataBlock (%): 70_ _2020-09-22 18:33:52,281 INFO org...: BlockCache evicted (MB): 5687, overhead (%): 1795, heavy eviction counter: 33, current caching DataBlock (%): 55_ _2020-09-22 18:34:03,394 INFO org...: BlockCache evicted (MB): 4136, overhead (%): 1278, heavy eviction counter: 34, current caching DataBlock (%): 43_ _2020-09-22 18:34:15,088 INFO org...: BlockCache evicted (MB): 2585, overhead (%): 761, heavy eviction counter: 35, current caching DataBlock (%): 36_ _2020-09-22 18:34:27,752 INFO org...: BlockCache evicted (MB): 1551, overhead (%): 417, heavy eviction counter: 36, current caching DataBlock (%): 32_ _2020-09-22 18:34:45,233 INFO org...: BlockCache evicted (MB): 940, overhead (%): 213, heavy eviction counter: 37, current caching DataBlock (%): 30_ _2020-09-22 18:34:55,364 INFO org...: BlockCache evicted (MB): 289, overhead (%): -4, heavy eviction counter: 37, current caching DataBlock (%): 31_ _2020-09-22 18:35:05,466 INFO org...: BlockCache evicted (MB): 240, overhead (%): -20, heavy eviction counter: 37, current caching DataBlock (%): 34_ _2020-09-22 18:35:15,564 INFO org...: BlockCache evicted (MB): 254, overhead (%): -16, heavy eviction counter: 37, current caching DataBlock (%): 36_ _2020-09-22 18:35:25,670 INFO org...: BlockCache evicted (MB): 279, overhead (%): -7, heavy eviction counter: 37, current caching DataBlock (%): 37_ _2020-09-22 18:35:35,801 INFO org...: BlockCache evicted (MB): 294, overhead (%): -2, heavy eviction counter: 37, current caching DataBlock (%): 38_ _2020-09-22 18:35:45,918 INFO org...: BlockCache evicted (MB): 309, overhead (%): 3, heavy eviction counter: 38, current caching DataBlock (%): 38_ _2020-09-22 18:35:56,027 INFO org...: BlockCache evicted (MB): 253, overhead (%): -16, heavy eviction counter: 38, current caching DataBlock (%): 40_
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200618#comment-17200618 ] Danil Lipovoy commented on HBASE-23887: --- I am going to write some article about this and have done some test CS vs HB. Think it could be interesting. I run YCSB from 2 hosts (800 threads summary) on tables which size: HBase - 300 GB on HDFS (100 GB pure data) Cassandra - 250 GB (replication factor = 3) It means the volume approximately the same (HB a little bit more). The HB parameters: _dfs.client.short.circuit.num = 5 - this is another my improvement https://issues.apache.org/jira/browse/HDFS-15202 - it helps to speed up HB more_ _hbase.lru.cache.heavy.eviction.count.limit = 30 - it means the patch will work after 30 evictions (~5 minutes)_ _hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - good target for eviction_ So, I aggregated logs YCSB and put this into Excel: !image-2020-09-23-10-06-11-189.png! At the beginning CS faster then HB. When _heavy.eviction.count.limit_ pass 30 was enables the improvement and performance become the same. How it looks into the log of RegionServer: _2020-09-22 18:31:47,561 INFO org...LruBlockCache: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 21, current caching DataBlock (%): 100_ _2020-09-22 18:31:57,808 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 22, current caching DataBlock (%): 100_ _2020-09-22 18:32:08,051 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 23, current caching DataBlock (%): 100_ _2020-09-22 18:32:18,155 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 24, current caching DataBlock (%): 100_ _2020-09-22 18:32:28,479 INFO org...LruBlockCache: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 25, current caching DataBlock (%): 100_ _2020-09-22 18:32:38,754 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 26, current caching DataBlock (%): 100_ _2020-09-22 18:32:49,334 INFO org...LruBlockCache: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 27, current caching DataBlock (%): 100_ _2020-09-22 18:32:59,712 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 28, current caching DataBlock (%): 100_ _2020-09-22 18:33:10,061 INFO org...LruBlockCache: BlockCache evicted (MB): 7238, overhead (%): 2312, heavy eviction counter: 29, current caching DataBlock (%): 100_ _2020-09-22 18:33:20,220 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 30, current caching DataBlock (%): 100 <- the feature enabled_ _2020-09-22 18:33:30,314 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 31, current caching DataBlock (%): 85_ _2020-09-22 18:33:41,390 INFO org...LruBlockCache: BlockCache evicted (MB): 6721, overhead (%): 2140, heavy eviction counter: 32, current caching DataBlock (%): 70_ _2020-09-22 18:33:52,281 INFO org...LruBlockCache: BlockCache evicted (MB): 5687, overhead (%): 1795, heavy eviction counter: 33, current caching DataBlock (%): 55_ _2020-09-22 18:34:03,394 INFO org...LruBlockCache: BlockCache evicted (MB): 4136, overhead (%): 1278, heavy eviction counter: 34, current caching DataBlock (%): 43_ _2020-09-22 18:34:15,088 INFO org...LruBlockCache: BlockCache evicted (MB): 2585, overhead (%): 761, heavy eviction counter: 35, current caching DataBlock (%): 36_ _2020-09-22 18:34:27,752 INFO org...LruBlockCache: BlockCache evicted (MB): 1551, overhead (%): 417, heavy eviction counter: 36, current caching DataBlock (%): 32_ _2020-09-22 18:34:45,233 INFO org...LruBlockCache: BlockCache evicted (MB): 940, overhead (%): 213, heavy eviction counter: 37, current caching DataBlock (%): 30_ _2020-09-22 18:34:55,364 INFO org...LruBlockCache: BlockCache evicted (MB): 289, overhead (%): -4, heavy eviction counter: 37, current caching DataBlock (%): 31_ _2020-09-22 18:35:05,466 INFO org...LruBlockCache: BlockCache evicted (MB): 240, overhead (%): -20, heavy eviction counter: 37, current caching DataBlock (%): 34_ _2020-09-22 18:35:15,564 INFO org...LruBlockCache: BlockCache evicted (MB): 254, overhead (%): -16, heavy eviction counter: 37, current caching DataBlock (%): 36_ _2020-09-22 18:35:25,670 INFO org...LruBlockCache: BlockCache evicted (MB): 279, overhead (%): -7, heavy eviction counter: 37, current caching DataBlock (%): 37_ _2020-09-22 18:35:35,801 INFO org...LruBlockCache: BlockCache evicted (MB): 294, overhead (%): -2, heavy eviction counter: 37, current caching DataBlock (%): 38_ _2020-09-22 18:35:45,918 INFO org...LruBlock
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: ycsb_logs.zip
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: image-2020-09-23-10-06-11-189.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200603#comment-17200603
]
Danil Lipovoy commented on HBASE-23887:
---
[~elserj] could you explain please, how to enable AdaptiveLruBlockCache ?
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, ratio.png, ratio2.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200601#comment-17200601
]
Danil Lipovoy commented on HBASE-23887:
---
[~vrodionov], I did simple test - disable BC and run YCSB on table 100 GB.
Without the cache is working very slow:
!image-2020-09-23-09-48-59-714.png!
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, ratio.png, ratio2.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: image-2020-09-23-09-48-59-714.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, ratio.png, ratio2.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166191#comment-17166191
]
Danil Lipovoy commented on HBASE-23887:
---
Bharath, all is fine, take your time) Please, let me know if I can do something
to merge this feature. I would be very proud to make some little contribution
into our lovely HBase.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165575#comment-17165575
]
Danil Lipovoy commented on HBASE-23887:
---
[~bharathv]
Looks like there is some obstacle, could you please explain, what is a concern?
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17155273#comment-17155273
]
Danil Lipovoy commented on HBASE-23887:
---
[~bharathv], can I help by some way to merge the feature?
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/22/20, 5:57 AM: - Is it ok for the summary doc? Sorry for a lot of mistakes, my english quite bad. Hope someone would correct the text. — Sometimes we are reading much more data than can fit into BlockCache and it is the cause of a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !image-2020-06-22-05-57-45-578.png! We could avoid this situation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times we have to run the eviction process that starts to avoid putting data to BlockCache. By default it is 2147483647 and actually equals to disable the feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to be about 1 minutes, then we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and want all of them in the cache we will have it (except for eviction of course). When we use long-term heavy reading the feature will be enabled after some time and bring better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintain it. Don't try to set it too small because it leads to premature exit from this mode. For powerful CPUs (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak systems (2-5 cores) may be good with 50-100 MB. How it works: we set the limit and after each ~10 second calculate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: _BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100_ < no eviction, 100% blocks is caching _BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97_ < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevents premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our reading is heavy for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we aren't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is the value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, then percent of caching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Maybe it is just short-term fluctuation and we will try to stay in this mode. It helps avoid premature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-18-35-48-366.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223, evict 198 A l
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135251#comment-17135251 ] Danil Lipovoy edited comment on HBASE-23887 at 6/14/20, 7:04 PM: - And one more test. Before there are two different tables for scan and other for gets. Now the table was the same: !scan_and_gets2.png! The ratio looks different because reading the same blocks. evicted (MB): 0, ratio 0.0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): 0, ratio 0.0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): 2170, ratio 1.09, overhead (%): 985, heavy eviction counter: 1, current caching DataBlock (%): 91 < start evicted (MB): 3763, ratio 1.08, overhead (%): 1781, heavy eviction counter: 2, current caching DataBlock (%): 76 evicted (MB): 3306, ratio 1.07, overhead (%): 1553, heavy eviction counter: 3, current caching DataBlock (%): 61 evicted (MB): 2508, ratio 1.06, overhead (%): 1154, heavy eviction counter: 4, current caching DataBlock (%): 50 evicted (MB): 1824, ratio 1.04, overhead (%): 812, heavy eviction counter: 5, current caching DataBlock (%): 42 evicted (MB): 1482, ratio 1.03, overhead (%): 641, heavy eviction counter: 6, current caching DataBlock (%): 36 evicted (MB): 1140, ratio 1.01, overhead (%): 470, heavy eviction counter: 7, current caching DataBlock (%): 32 evicted (MB): 913, ratio 1.0, overhead (%): 356, heavy eviction counter: 8, current caching DataBlock (%): 29 evicted (MB): 912, ratio 0.89, overhead (%): 356, heavy eviction counter: 9, current caching DataBlock (%): 26 evicted (MB): 684, ratio 0.76, overhead (%): 242, heavy eviction counter: 10, current caching DataBlock (%): 24 evicted (MB): 684, ratio 0.61, overhead (%): 242, heavy eviction counter: 11, current caching DataBlock (%): 22 evicted (MB): 456, ratio 0.51, overhead (%): 128, heavy eviction counter: 12, current caching DataBlock (%): 21 evicted (MB): 456, ratio 0.42, overhead (%): 128, heavy eviction counter: 13, current caching DataBlock (%): 20 evicted (MB): 456, ratio 0.33, overhead (%): 128, heavy eviction counter: 14, current caching DataBlock (%): 19 evicted (MB): 342, ratio 0.33, overhead (%): 71, heavy eviction counter: 15, current caching DataBlock (%): 19 evicted (MB): 342, ratio 0.32, overhead (%): 71, heavy eviction counter: 16, current caching DataBlock (%): 19 evicted (MB): 342, ratio 0.31, overhead (%): 71, heavy eviction counter: 17, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.3, overhead (%): 14, heavy eviction counter: 18, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.29, overhead (%): 14, heavy eviction counter: 19, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.27, overhead (%): 14, heavy eviction counter: 20, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.25, overhead (%): 14, heavy eviction counter: 21, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.24, overhead (%): 14, heavy eviction counter: 22, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.22, overhead (%): 14, heavy eviction counter: 23, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.21, overhead (%): 14, heavy eviction counter: 24, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.2, overhead (%): 14, heavy eviction counter: 25, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.17, overhead (%): 14, heavy eviction counter: 26, current caching DataBlock (%): 19 evicted (MB): 456, ratio 0.17, overhead (%): 128, heavy eviction counter: 27, current caching DataBlock (%): 18 < added gets (but table the same) evicted (MB): 456, ratio 0.15, overhead (%): 128, heavy eviction counter: 28, current caching DataBlock (%): 17 evicted (MB): 342, ratio 0.13, overhead (%): 71, heavy eviction counter: 29, current caching DataBlock (%): 17 evicted (MB): 342, ratio 0.11, overhead (%): 71, heavy eviction counter: 30, current caching DataBlock (%): 17 evicted (MB): 342, ratio 0.09, overhead (%): 71, heavy eviction counter: 31, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.08, overhead (%): 14, heavy eviction counter: 32, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.07, overhead (%): 14, heavy eviction counter: 33, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.06, overhead (%): 14, heavy eviction counter: 34, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.05, overhead (%): 14, heavy eviction counter: 35, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.05, overhead (%): 14, heavy eviction counter: 36, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.04, overhead (%): 14, heavy eviction counter: 37, current caching DataBlock (%): 17 evicted (MB): 109, ratio 0.04,
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135251#comment-17135251 ] Danil Lipovoy commented on HBASE-23887: --- And one more test. Before there are two different tables for scan and other for gets. Now the table was the same: !scan_and_gets2.png! !image-2020-06-14-20-51-11-905.png! The ratio looks different because reading the same blocks. evicted (MB): 0, ratio 0.0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): 0, ratio 0.0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): 2170, ratio 1.09, overhead (%): 985, heavy eviction counter: 1, current caching DataBlock (%): 91 < start evicted (MB): 3763, ratio 1.08, overhead (%): 1781, heavy eviction counter: 2, current caching DataBlock (%): 76 evicted (MB): 3306, ratio 1.07, overhead (%): 1553, heavy eviction counter: 3, current caching DataBlock (%): 61 evicted (MB): 2508, ratio 1.06, overhead (%): 1154, heavy eviction counter: 4, current caching DataBlock (%): 50 evicted (MB): 1824, ratio 1.04, overhead (%): 812, heavy eviction counter: 5, current caching DataBlock (%): 42 evicted (MB): 1482, ratio 1.03, overhead (%): 641, heavy eviction counter: 6, current caching DataBlock (%): 36 evicted (MB): 1140, ratio 1.01, overhead (%): 470, heavy eviction counter: 7, current caching DataBlock (%): 32 evicted (MB): 913, ratio 1.0, overhead (%): 356, heavy eviction counter: 8, current caching DataBlock (%): 29 evicted (MB): 912, ratio 0.89, overhead (%): 356, heavy eviction counter: 9, current caching DataBlock (%): 26 evicted (MB): 684, ratio 0.76, overhead (%): 242, heavy eviction counter: 10, current caching DataBlock (%): 24 evicted (MB): 684, ratio 0.61, overhead (%): 242, heavy eviction counter: 11, current caching DataBlock (%): 22 evicted (MB): 456, ratio 0.51, overhead (%): 128, heavy eviction counter: 12, current caching DataBlock (%): 21 evicted (MB): 456, ratio 0.42, overhead (%): 128, heavy eviction counter: 13, current caching DataBlock (%): 20 evicted (MB): 456, ratio 0.33, overhead (%): 128, heavy eviction counter: 14, current caching DataBlock (%): 19 evicted (MB): 342, ratio 0.33, overhead (%): 71, heavy eviction counter: 15, current caching DataBlock (%): 19 evicted (MB): 342, ratio 0.32, overhead (%): 71, heavy eviction counter: 16, current caching DataBlock (%): 19 evicted (MB): 342, ratio 0.31, overhead (%): 71, heavy eviction counter: 17, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.3, overhead (%): 14, heavy eviction counter: 18, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.29, overhead (%): 14, heavy eviction counter: 19, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.27, overhead (%): 14, heavy eviction counter: 20, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.25, overhead (%): 14, heavy eviction counter: 21, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.24, overhead (%): 14, heavy eviction counter: 22, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.22, overhead (%): 14, heavy eviction counter: 23, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.21, overhead (%): 14, heavy eviction counter: 24, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.2, overhead (%): 14, heavy eviction counter: 25, current caching DataBlock (%): 19 evicted (MB): 228, ratio 0.17, overhead (%): 14, heavy eviction counter: 26, current caching DataBlock (%): 19 evicted (MB): 456, ratio 0.17, overhead (%): 128, heavy eviction counter: 27, current caching DataBlock (%): 18 < added gets (but table the same) evicted (MB): 456, ratio 0.15, overhead (%): 128, heavy eviction counter: 28, current caching DataBlock (%): 17 evicted (MB): 342, ratio 0.13, overhead (%): 71, heavy eviction counter: 29, current caching DataBlock (%): 17 evicted (MB): 342, ratio 0.11, overhead (%): 71, heavy eviction counter: 30, current caching DataBlock (%): 17 evicted (MB): 342, ratio 0.09, overhead (%): 71, heavy eviction counter: 31, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.08, overhead (%): 14, heavy eviction counter: 32, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.07, overhead (%): 14, heavy eviction counter: 33, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.06, overhead (%): 14, heavy eviction counter: 34, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.05, overhead (%): 14, heavy eviction counter: 35, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.05, overhead (%): 14, heavy eviction counter: 36, current caching DataBlock (%): 17 evicted (MB): 228, ratio 0.04, overhead (%): 14, heavy eviction counter: 37, current caching DataBlock (%): 17 evicted (MB): 109, ratio 0.04, overhead (%): -46, heavy eviction counter: 37, curren
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: ratio2.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, ratio.png, ratio2.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: image-2020-06-14-20-51-11-905.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, ratio.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: scan_and_gets2.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> ratio.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135248#comment-17135248
]
Danil Lipovoy commented on HBASE-23887:
---
Plus picture:
!ratio.png!
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> ratio.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: ratio.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> ratio.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135184#comment-17135184 ] Danil Lipovoy edited comment on HBASE-23887 at 6/14/20, 2:51 PM: - And the log the second try (the feature is enabled): evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -905, ratio 0.0, overhead (%): -552, heavy eviction counter: 0, current caching DataBlock (%): 100 < start the test evicted (MB): 4676, ratio 1.07, overhead (%): 2238, heavy eviction counter: 1, current caching DataBlock (%): 85 evicted (MB): 4561, ratio 1.05, overhead (%): 2180, heavy eviction counter: 2, current caching DataBlock (%): 70 evicted (MB): 3535, ratio 1.04, overhead (%): 1667, heavy eviction counter: 3, current caching DataBlock (%): 55 evicted (MB): 2508, ratio 1.02, overhead (%): 1154, heavy eviction counter: 4, current caching DataBlock (%): 44 evicted (MB): 1824, ratio 0.88, overhead (%): 812, heavy eviction counter: 5, current caching DataBlock (%): 36 evicted (MB): 1255, ratio 0.61, overhead (%): 527, heavy eviction counter: 6, current caching DataBlock (%): 31 evicted (MB): 912, ratio 0.41, overhead (%): 356, heavy eviction counter: 7, current caching DataBlock (%): 28 evicted (MB): 684, ratio 0.32, overhead (%): 242, heavy eviction counter: 8, current caching DataBlock (%): 26 evicted (MB): 570, ratio 0.29, overhead (%): 185, heavy eviction counter: 9, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.24, overhead (%): 71, heavy eviction counter: 10, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 11, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 12, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 13, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 14, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.08, overhead (%): 14, heavy eviction counter: 15, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.07, overhead (%): 14, heavy eviction counter: 16, current caching DataBlock (%): 25 evicted (MB): 223, ratio 0.06, overhead (%): 11, heavy eviction counter: 17, current caching DataBlock (%): 25 evicted (MB): 107, ratio 0.06, overhead (%): -47, heavy eviction counter: 17, current caching DataBlock (%): 30 < back pressure evicted (MB): 456, ratio 0.16, overhead (%): 128, heavy eviction counter: 18, current caching DataBlock (%): 29 evicted (MB): 456, ratio 0.19, overhead (%): 128, heavy eviction counter: 19, current caching DataBlock (%): 28 evicted (MB): 456, ratio 0.2, overhead (%): 128, heavy eviction counter: 20, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 21, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.17, overhead (%): 71, heavy eviction counter: 22, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.16, overhead (%): 71, heavy eviction counter: 23, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 24, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.13, overhead (%): 71, heavy eviction counter: 25, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 26, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 27, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 28, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 29, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 30, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 31, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 32, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.09, overhead (%): 14, heavy eviction counter: 33, current caching DataBlock (%): 27 evicted (MB): 1026, ratio 0.42, overhead (%): 413, heavy eviction counter: 34, current caching DataBlock (%): 23 < added gets evicted (MB): 1140, ratio 0.66, overhead (%): 470, heavy eviction counter: 35, current caching DataBlock (%): 19 evicted (MB): 913, ratio 0.75, overhead (
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135184#comment-17135184 ] Danil Lipovoy edited comment on HBASE-23887 at 6/14/20, 2:49 PM: - And the log the second try (the feature is enabled): evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -905, ratio 0.0, overhead (%): -552, heavy eviction counter: 0, current caching DataBlock (%): 100 < start the test evicted (MB): 4676, ratio 1.07, overhead (%): 2238, heavy eviction counter: 1, current caching DataBlock (%): 85 evicted (MB): 4561, ratio 1.05, overhead (%): 2180, heavy eviction counter: 2, current caching DataBlock (%): 70 evicted (MB): 3535, ratio 1.04, overhead (%): 1667, heavy eviction counter: 3, current caching DataBlock (%): 55 evicted (MB): 2508, ratio 1.02, overhead (%): 1154, heavy eviction counter: 4, current caching DataBlock (%): 44 evicted (MB): 1824, ratio 0.88, overhead (%): 812, heavy eviction counter: 5, current caching DataBlock (%): 36 evicted (MB): 1255, ratio 0.61, overhead (%): 527, heavy eviction counter: 6, current caching DataBlock (%): 31 evicted (MB): 912, ratio 0.41, overhead (%): 356, heavy eviction counter: 7, current caching DataBlock (%): 28 evicted (MB): 684, ratio 0.32, overhead (%): 242, heavy eviction counter: 8, current caching DataBlock (%): 26 evicted (MB): 570, ratio 0.29, overhead (%): 185, heavy eviction counter: 9, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.24, overhead (%): 71, heavy eviction counter: 10, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 11, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 12, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 13, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 14, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.08, overhead (%): 14, heavy eviction counter: 15, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.07, overhead (%): 14, heavy eviction counter: 16, current caching DataBlock (%): 25 evicted (MB): 223, ratio 0.06, overhead (%): 11, heavy eviction counter: 17, current caching DataBlock (%): 25 evicted (MB): 107, ratio 0.06, overhead (%): -47, heavy eviction counter: 17, current caching DataBlock (%): 30 < back pressure evicted (MB): 456, ratio 0.16, overhead (%): 128, heavy eviction counter: 18, current caching DataBlock (%): 29 evicted (MB): 456, ratio 0.19, overhead (%): 128, heavy eviction counter: 19, current caching DataBlock (%): 28 evicted (MB): 456, ratio 0.2, overhead (%): 128, heavy eviction counter: 20, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 21, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.17, overhead (%): 71, heavy eviction counter: 22, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.16, overhead (%): 71, heavy eviction counter: 23, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 24, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.13, overhead (%): 71, heavy eviction counter: 25, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 26, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 27, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 28, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 29, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 30, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 31, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 32, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.09, overhead (%): 14, heavy eviction counter: 33, current caching DataBlock (%): 27 evicted (MB): 1026, ratio 0.42, overhead (%): 413, heavy eviction counter: 34, current caching DataBlock (%): 23 < added gets evicted (MB): 1140, ratio 0.66, overhead (%): 470, heavy eviction counter: 35, current caching DataBlock (%): 19 evicted (MB): 913, ratio 0.75, overhead (
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135181#comment-17135181 ] Danil Lipovoy edited comment on HBASE-23887 at 6/14/20, 2:47 PM: - [~bharathv] I found out why scan was slow - the bottle neck was network (I sent requests form other PC). Now I scan local and the network is not the problem. So I run a test scenario: 1. Scan (25 threads, batch = 100) 2. After 5 minutes add multi-gets (25 threads, batch = 100) 3. After 5 minutes switch off multi-gets (only scan again) During the test have checked the ratio between single and multi caches (added it into log file). The first run with *hbase.lru.cache.heavy.eviction.count.limit* = 1 (disable the feature) and the second the limit = 0. !scan_and_gets.png! Take a look on the ratio (single/multi). Log file of the first run (the feature is disabled): evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): 114, ratio 5.78, overhead (%): -43, heavy eviction counter: 0, current caching DataBlock (%): 100 < start the test evicted (MB): 5704, ratio 1.07, overhead (%): 2752, heavy eviction counter: 1, current caching DataBlock (%): 100 evicted (MB): 5245, ratio 1.06, overhead (%): 2522, heavy eviction counter: 2, current caching DataBlock (%): 100 evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 3, current caching DataBlock (%): 100 evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter: 4, current caching DataBlock (%): 100 evicted (MB): 5132, ratio 1.06, overhead (%): 2466, heavy eviction counter: 5, current caching DataBlock (%): 100 evicted (MB): 5018, ratio 1.07, overhead (%): 2409, heavy eviction counter: 6, current caching DataBlock (%): 100 evicted (MB): 5244, ratio 1.06, overhead (%): 2522, heavy eviction counter: 7, current caching DataBlock (%): 100 evicted (MB): 5019, ratio 1.07, overhead (%): 2409, heavy eviction counter: 8, current caching DataBlock (%): 100 evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 9, current caching DataBlock (%): 100 evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter: 10, current caching DataBlock (%): 100 evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter: 11, current caching DataBlock (%): 100 evicted (MB): 4563, ratio 1.06, overhead (%): 2181, heavy eviction counter: 12, current caching DataBlock (%): 100 evicted (MB): 4338, ratio 1.06, overhead (%): 2069, heavy eviction counter: 13, current caching DataBlock (%): 100 evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter: 14, current caching DataBlock (%): 100 evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 15, current caching DataBlock (%): 100 evicted (MB): 5130, ratio 1.06, overhead (%): 2465, heavy eviction counter: 16, current caching DataBlock (%): 100 evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter: 17, current caching DataBlock (%): 100 evicted (MB): 4795, ratio 1.06, overhead (%): 2297, heavy eviction counter: 18, current caching DataBlock (%): 100 evicted (MB): 4905, ratio 1.07, overhead (%): 2352, heavy eviction counter: 19, current caching DataBlock (%): 100 evicted (MB): 4911, ratio 1.06, overhead (%): 2355, heavy eviction counter: 20, current caching DataBlock (%): 100 evicted (MB): 5019, ratio 1.06, overhead (%): 2409, heavy eviction counter: 21, current caching DataBlock (%): 100 evicted (MB): 5134, ratio 1.06, overhead (%): 2467, heavy eviction counter: 22, current caching DataBlock (%): 100 evicted (MB): 5016, ratio 1.06, overhead (%): 2408, heavy eviction counter: 23, current caching DataBlock (%): 100 evicted (MB): 4450, ratio 1.06, overhead (%): 2125, heavy eviction counter: 24, current caching DataBlock (%): 100 evicted (MB): 4904, ratio 1.07, overhead (%): 2352, heavy eviction counter: 25, current caching DataBlock (%): 100 evicted (MB): 4561, ratio 1.06, overhead (%): 2180, heavy eviction counter: 26, current caching DataBlock (%): 100 evicted (MB): 4334, ratio 1.06, overhead (%): 2067, heavy eviction counter: 27, current caching DataBlock (%): 100 evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter: 28, current caching DataBlock (%): 100 evicted (MB): 4792, ratio 1.06, overhead (%): 2296, heavy eviction counter: 29, current caching DataBlock (%): 100 evicted (MB): 4903, ratio 1.07, overhead (%): 2351, heavy eviction counter: 30, current caching DataBlock (%): 100 evicted (MB): 4791,
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135184#comment-17135184 ] Danil Lipovoy commented on HBASE-23887: --- And the log the second try (the feature is enabled): evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -905, ratio 0.0, overhead (%): -552, heavy eviction counter: 0, current caching DataBlock (%): 100 < start the test evicted (MB): 4676, ratio 1.07, overhead (%): 2238, heavy eviction counter: 1, current caching DataBlock (%): 85 evicted (MB): 4561, ratio 1.05, overhead (%): 2180, heavy eviction counter: 2, current caching DataBlock (%): 70 evicted (MB): 3535, ratio 1.04, overhead (%): 1667, heavy eviction counter: 3, current caching DataBlock (%): 55 evicted (MB): 2508, ratio 1.02, overhead (%): 1154, heavy eviction counter: 4, current caching DataBlock (%): 44 evicted (MB): 1824, ratio 0.88, overhead (%): 812, heavy eviction counter: 5, current caching DataBlock (%): 36 evicted (MB): 1255, ratio 0.61, overhead (%): 527, heavy eviction counter: 6, current caching DataBlock (%): 31 evicted (MB): 912, ratio 0.41, overhead (%): 356, heavy eviction counter: 7, current caching DataBlock (%): 28 evicted (MB): 684, ratio 0.32, overhead (%): 242, heavy eviction counter: 8, current caching DataBlock (%): 26 evicted (MB): 570, ratio 0.29, overhead (%): 185, heavy eviction counter: 9, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.24, overhead (%): 71, heavy eviction counter: 10, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 11, current caching DataBlock (%): 25 evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 12, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 13, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 14, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.08, overhead (%): 14, heavy eviction counter: 15, current caching DataBlock (%): 25 evicted (MB): 228, ratio 0.07, overhead (%): 14, heavy eviction counter: 16, current caching DataBlock (%): 25 evicted (MB): 223, ratio 0.06, overhead (%): 11, heavy eviction counter: 17, current caching DataBlock (%): 25 evicted (MB): 107, ratio 0.06, overhead (%): -47, heavy eviction counter: 17, current caching DataBlock (%): 30 < back pressure evicted (MB): 456, ratio 0.16, overhead (%): 128, heavy eviction counter: 18, current caching DataBlock (%): 29 evicted (MB): 456, ratio 0.19, overhead (%): 128, heavy eviction counter: 19, current caching DataBlock (%): 28 evicted (MB): 456, ratio 0.2, overhead (%): 128, heavy eviction counter: 20, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.19, overhead (%): 71, heavy eviction counter: 21, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.17, overhead (%): 71, heavy eviction counter: 22, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.16, overhead (%): 71, heavy eviction counter: 23, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.14, overhead (%): 71, heavy eviction counter: 24, current caching DataBlock (%): 27 evicted (MB): 342, ratio 0.13, overhead (%): 71, heavy eviction counter: 25, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 26, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.12, overhead (%): 14, heavy eviction counter: 27, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 28, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.11, overhead (%): 14, heavy eviction counter: 29, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 30, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 31, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.1, overhead (%): 14, heavy eviction counter: 32, current caching DataBlock (%): 27 evicted (MB): 228, ratio 0.09, overhead (%): 14, heavy eviction counter: 33, current caching DataBlock (%): 27 evicted (MB): 1026, ratio 0.42, overhead (%): 413, heavy eviction counter: 34, current caching DataBlock (%): 23 < added gets evicted (MB): 1140, ratio 0.66, overhead (%): 470, heavy eviction counter: 35, current caching DataBlock (%): 19 evicted (MB): 913, ratio 0.75, overhead (%): 356, heavy eviction counter: 36, current caching DataBlock (%): 16 evicted (MB): 798,
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135181#comment-17135181 ] Danil Lipovoy commented on HBASE-23887: --- [~bharathv] I found out why scan was slow - the bottle neck was net (I sent requests form other PC). Now I scan local and the net is not the problem. So I run a test scenario: 1. Scan (25 threads, batch = 100) 2. After 5 minutes add multi-gets (25 threads, batch = 100) 3. After 5 minutes switch off multi-gets (only scan again) During the test have checked the ratio between single and multi caches (added it into log file). The first run with *hbase.lru.cache.heavy.eviction.count.limit* = 1 (disable the feature) and the second the limit = 0. !scan_and_gets.png! Take a look on the ratio (single/multi). Log file of the first run (the feature is disabled): evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): -2721, ratio 0.0, overhead (%): -1460, heavy eviction counter: 0, current caching DataBlock (%): 100 evicted (MB): 114, ratio 5.78, overhead (%): -43, heavy eviction counter: 0, current caching DataBlock (%): 100 < start the test evicted (MB): 5704, ratio 1.07, overhead (%): 2752, heavy eviction counter: 1, current caching DataBlock (%): 100 evicted (MB): 5245, ratio 1.06, overhead (%): 2522, heavy eviction counter: 2, current caching DataBlock (%): 100 evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 3, current caching DataBlock (%): 100 evicted (MB): 4788, ratio 1.06, overhead (%): 2294, heavy eviction counter: 4, current caching DataBlock (%): 100 evicted (MB): 5132, ratio 1.06, overhead (%): 2466, heavy eviction counter: 5, current caching DataBlock (%): 100 evicted (MB): 5018, ratio 1.07, overhead (%): 2409, heavy eviction counter: 6, current caching DataBlock (%): 100 evicted (MB): 5244, ratio 1.06, overhead (%): 2522, heavy eviction counter: 7, current caching DataBlock (%): 100 evicted (MB): 5019, ratio 1.07, overhead (%): 2409, heavy eviction counter: 8, current caching DataBlock (%): 100 evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 9, current caching DataBlock (%): 100 evicted (MB): 4904, ratio 1.06, overhead (%): 2352, heavy eviction counter: 10, current caching DataBlock (%): 100 evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter: 11, current caching DataBlock (%): 100 evicted (MB): 4563, ratio 1.06, overhead (%): 2181, heavy eviction counter: 12, current caching DataBlock (%): 100 evicted (MB): 4338, ratio 1.06, overhead (%): 2069, heavy eviction counter: 13, current caching DataBlock (%): 100 evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter: 14, current caching DataBlock (%): 100 evicted (MB): 4902, ratio 1.06, overhead (%): 2351, heavy eviction counter: 15, current caching DataBlock (%): 100 evicted (MB): 5130, ratio 1.06, overhead (%): 2465, heavy eviction counter: 16, current caching DataBlock (%): 100 evicted (MB): 5017, ratio 1.06, overhead (%): 2408, heavy eviction counter: 17, current caching DataBlock (%): 100 evicted (MB): 4795, ratio 1.06, overhead (%): 2297, heavy eviction counter: 18, current caching DataBlock (%): 100 evicted (MB): 4905, ratio 1.07, overhead (%): 2352, heavy eviction counter: 19, current caching DataBlock (%): 100 evicted (MB): 4911, ratio 1.06, overhead (%): 2355, heavy eviction counter: 20, current caching DataBlock (%): 100 evicted (MB): 5019, ratio 1.06, overhead (%): 2409, heavy eviction counter: 21, current caching DataBlock (%): 100 evicted (MB): 5134, ratio 1.06, overhead (%): 2467, heavy eviction counter: 22, current caching DataBlock (%): 100 evicted (MB): 5016, ratio 1.06, overhead (%): 2408, heavy eviction counter: 23, current caching DataBlock (%): 100 evicted (MB): 4450, ratio 1.06, overhead (%): 2125, heavy eviction counter: 24, current caching DataBlock (%): 100 evicted (MB): 4904, ratio 1.07, overhead (%): 2352, heavy eviction counter: 25, current caching DataBlock (%): 100 evicted (MB): 4561, ratio 1.06, overhead (%): 2180, heavy eviction counter: 26, current caching DataBlock (%): 100 evicted (MB): 4334, ratio 1.06, overhead (%): 2067, heavy eviction counter: 27, current caching DataBlock (%): 100 evicted (MB): 4789, ratio 1.06, overhead (%): 2294, heavy eviction counter: 28, current caching DataBlock (%): 100 evicted (MB): 4792, ratio 1.06, overhead (%): 2296, heavy eviction counter: 29, current caching DataBlock (%): 100 evicted (MB): 4903, ratio 1.07, overhead (%): 2351, heavy eviction counter: 30, current caching DataBlock (%): 100 evicted (MB): 4791, ratio 1.06, overhead (%): 2295, heavy eviction counter: 31, current caching DataBlock (%):
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: scan_and_gets.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/10/20, 5:40 AM: - Is it ok for the summary doc? Sorry for a lot of mistakes, my english quite bad. Hope someone would correct the text. — Sometimes we are reading much more data than can fit into BlockCache and it is the cause of a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! (I will actualize the name of param in the gif later) We could avoid this situation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times we have to run the eviction process that starts to avoid putting data to BlockCache. By default it is 2147483647 and actually equals to disable the feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to be about 1 minutes, then we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and want all of them in the cache we will have it (except for eviction of course). When we use long-term heavy reading the feature will be enabled after some time and bring better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintain it. Don't try to set it too small because it leads to premature exit from this mode. For powerful CPUs (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak systems (2-5 cores) may be good with 50-100 MB. How it works: we set the limit and after each ~10 second calculate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: _BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100_ < no eviction, 100% blocks is caching _BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97_ < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevents premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our reading is heavy for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we aren't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is the value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, then percent of caching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Maybe it is just short-term fluctuation and we will try to stay in this mode. It helps avoid premature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-18-35-48-366.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 12
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Description:
Hi!
I first time here, correct me please if something wrong.
All latest information is here:
[https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
I want propose how to improve performance when data in HFiles much more than
BlockChache (usual story in BigData). The idea - caching only part of DATA
blocks. It is good becouse LruBlockCache starts to work and save huge amount of
GC.
Sometimes we have more data than can fit into BlockCache and it is cause a high
rate of evictions. In this case we can skip cache a block N and insted cache
the N+1th block. Anyway we would evict N block quite soon and that why that
skipping good for performance.
---
Some information below isn't actual
---
Example:
Imagine we have little cache, just can fit only 1 block and we are trying to
read 3 blocks with offsets:
124
198
223
Current way - we put the block 124, then put 198, evict 124, put 223, evict
198. A lot of work (5 actions).
With the feature - last few digits evenly distributed from 0 to 99. When we
divide by modulus we got:
124 -> 24
198 -> 98
223 -> 23
It helps to sort them. Some part, for example below 50 (if we set
*hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip others.
It means we will not try to handle the block 198 and save CPU for other job. In
the result - we put block 124, then put 223, evict 124 (3 actions).
See the picture in attachment with test below. Requests per second is higher,
GC is lower.
The key point of the code:
Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
100
But if we set it 1-99, then will work the next logic:
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
return;
...
// the same code as usual
}
{code}
Other parameters help to control when this logic will be enabled. It means it
will work only while heavy reading going on.
hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
eviction process that start to avoid of putting data to BlockCache
hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
evicted each time that start to avoid of putting data to BlockCache
By default: if 10 times (100 secunds) evicted more than 10 MB (each time) then
we start to skip 50% of data blocks.
When heavy evitions process end then new logic off and will put into
BlockCache all blocks again.
Descriptions of the test:
4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
4 RegionServers
4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
Total BlockCache Size = 48 Gb (8 % of data in HFiles)
Random read in 20 threads
I am going to make Pull Request, hope it is right way to make some contribution
in this cool product.
was:
Hi!
I first time here, correct me please if something wrong.
I want propose how to improve performance when data in HFiles much more than
BlockChache (usual story in BigData). The idea - caching only part of DATA
blocks. It is good becouse LruBlockCache starts to work and save huge amount of
GC.
Sometimes we have more data than can fit into BlockCache and it is cause a high
rate of evictions. In this case we can skip cache a block N and insted cache
the N+1th block. Anyway we would evict N block quite soon and that why that
skipping good for performance.
Example:
Imagine we have little cache, just can fit only 1 block and we are trying to
read 3 blocks with offsets:
124
198
223
Current way - we put the block 124, then put 198, evict 124, put 223, evict
198. A lot of work (5 actions).
With the feature - last few digits evenly distributed from 0 to 99. When we
divide by modulus we got:
124 -> 24
198 -> 98
223 -> 23
It helps to sort them. Some part, for example below 50 (if we set
*hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip others.
It means we will not try to handle the block 198 and save CPU for other job. In
the result - we put block 124, then put 223, evict 124 (3 actions).
See the picture in attachment with test below. Requests per second is higher,
GC is lower.
The key point of the code:
Added the parameter: *hbase.lru.cache.data.block.percent* which by default = 100
But if we set it 1-99, then will work the next logic:
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
return;
...
// the same cod
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129400#comment-17129400
]
Danil Lipovoy commented on HBASE-23887:
---
Put it here:
https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/8/20, 3:38 PM: Is it ok for the summary doc? Sorry for a lot of mistakes, my english quite bad. Hope someone would correct the text. — Sometimes we are reading much more data than can fit into BlockCache and it is the cause a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! (I will actualize the name of param in the gif later) We could avoid this sitiuation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times have to run eviction process that start to avoid of putting data to BlockCache. By default it is 2147483647 and actually equals to disable feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to about 1 minutes, than we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and wanted all of them it the cache we will have it (except of evicted of course). When we use long-term heavy reading the featue will enabled after some time and bring better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintan it. Don't try to set it too small because it lead to premature exit from this mode. For powerful CPU (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak system (2-5 cores) maybe good with 50-100 MB. How it works: we set the limit and after each ~10 second caluclate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: _BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100_ < no eviction, 100% blocks is caching _BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97_ < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevent premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our heavy reading for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we don't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, than percent of chaching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Mayby it is just short-term fluctuation and we will try to stay in this mode. It help avoid permature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-18-35-48-366.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223,
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/8/20, 3:36 PM: Is it ok for the summury doc? — Sometimes we are reading much more data than can fit into BlockCache and it is the cause a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! (I will actualize the name of param in the gif later) We could avoid this sitiuation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times have to run eviction process that start to avoid of putting data to BlockCache. By default it is 2147483647 and actually equals to disable feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to about 1 minutes, than we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and wanted all of them it the cache we will have it (except of evicted of course). When we use long-term heavy reading the featue will enabled after some time and bring better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintan it. Don't try to set it too small because it lead to premature exit from this mode. For powerful CPU (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak system (2-5 cores) maybe good with 50-100 MB. How it works: we set the limit and after each ~10 second caluclate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: _BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100_ < no eviction, 100% blocks is caching _BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97_ < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevent premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our heavy reading for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we don't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, than percent of chaching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Mayby it is just short-term fluctuation and we will try to stay in this mode. It help avoid permature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-18-35-48-366.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223, evict 198 A lot of work (5 actions and 2 evictions). With the feature and *hbase.lru
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/8/20, 3:29 PM: Is it ok for the summury doc? — Sometimes we are reading much more data than can fit into BlockCache and it is the cause a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! (I will actualize the name of param in the gif later) We could avoid this sitiuation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times have to run eviction process that start to avoid of putting data to BlockCache. By default it is 2147483647 and actually equals to disable feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to about 1 minutes, than we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and wanted all of them it the cache we will have it (except of evicted of course). When we use long-term heavy reading the featue will enabled after some time and bring better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintan it. Don't try to set it too small because it lead to premature exit from this mode. For powerful CPU (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak system (2-5 cores) maybe good with 50-100 MB. How it works: we set the limit and after each ~10 second caluclate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: _BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100_ < no eviction, 100% blocks is caching _BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97_ < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevent premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our heavy reading for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we don't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, than percent of chaching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Mayby it is just short-term fluctuation and we will ty to stay in this mode. It help avoid permature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-17-38-52-579.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223, evict 198 A lot of work (5 actions and 2 evictions). With the feature and *hbase.lru.
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/8/20, 3:23 PM: Is it ok for the summury doc? — Sometimes we are reading much more data than can fit into BlockCache and it is the cause a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! (I will actualize the name of param in the gif later) We could avoid this sitiuation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times have to run eviction process that avoid of putting data to BlockCache. By default it is 2147483647 and actually equals to disable feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to about 1 minutes, than we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and wanted all of them it the cache we will have it (except of evicted of course). When we use long-term heavy reading the featue will enabled after some time and bring better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintan it. Don't try to set it too small because it lead to premature exit from this mode. For powerful CPU (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak system (2-5 cores) maybe good with 50-100 MB. How it works: we set the limit and after each ~10 second caluclate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 < no eviction, 100% blocks is caching BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97 < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevent premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our heavy reading for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we don't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, than percent of chaching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Mayby it is just short-term fluctuation and we will ty to stay in this mode. It help avoid permature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-17-38-52-579.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223, evict 198 A lot of work (5 actions and 2 evictions). With the feature and hbase.lru.cache.heavy.ev
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/8/20, 2:42 PM: Is it ok for the summury doc? — Sometimes we are reading more data than can fit into BlockCache and it is the cause a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! (I will actualize the name of param in the gif later) We could avoid this sitiuation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times have to run eviction process that avoid of putting data to BlockCache. By default it is 2147483647 and actually equals to disable feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to about 1 minutes, than we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and wanted all of them it the cache we will have it (except of evicted of course). When we use long-term heavy reading the featue will enabled after some time and birng better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintan it. Don't try to set it too small because it lead to premature exit from this mode. For powerful CPU (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak system (2-5 cores) maybe good with 50-100 MB. How it works: we set the limit and after each ~10 second caluclate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 < no eviction, 100% blocks is caching BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97 < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevent premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our heavy reading for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we don't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, than percent of chaching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Mayby it is just short-term fluctuation and we will ty to stay in this mode. It help avoid permature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-17-38-52-579.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223, evict 198 A lot of work (5 actions and 2 evictions). With the feature and hbase.lru.cache.heavy.evictio
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy edited comment on HBASE-23887 at 6/8/20, 2:38 PM: Is it ok for the summury doc? — Sometimes we are reading more data than can fit into BlockCache and it is the cause a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! We could avoid this sitiuation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times have to run eviction process that avoid of putting data to BlockCache. By default it is 2147483647 and actually equals to disable feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to about 1 minutes, than we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and wanted all of them it the cache we will have it (except of evicted of course). When we use long-term heavy reading the featue will enabled after some time and birng better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintan it. Don't try to set it too small because it lead to premature exit from this mode. For powerful CPU (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak system (2-5 cores) maybe good with 50-100 MB. How it works: we set the limit and after each ~10 second caluclate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 < no eviction, 100% blocks is caching BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97 < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevent premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our heavy reading for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we don't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, than percent of chaching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Mayby it is just short-term fluctuation and we will ty to stay in this mode. It help avoid permature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !image-2020-06-08-17-38-52-579.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223, evict 198 A lot of work (5 actions and 2 evictions). With the feature and hbase.lru.cache.heavy.eviction.count.limit = 0 and the auto-scaling have reached to
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128349#comment-17128349 ] Danil Lipovoy commented on HBASE-23887: --- Is it ok for the summury doc? --- Sometimes we are reading more data than can fit into BlockCache and it is the cause a high rate of evictions. This in turn leads to heavy Garbage Collector works. So a lot of blocks put into BlockCache but never read, but spending a lot of CPU resources for cleaning. !BlockCacheEvictionProcess.gif! We could avoid this sitiuation via parameters: *hbase.lru.cache.heavy.eviction.count.limit* - set how many times have to run eviction process that avoid of putting data to BlockCache. By default it is 2147483647 and actually equals to disable feature of increasing performance. Because eviction runs about every 5 - 10 second (it depends of workload) and 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 years. Just after that time it will start to work. We can set this parameter to 0 and get working the feature right now. But if we have some times short reading the same data and some times long-term reading - we can divide it by this parameter. For example we know that our short reading used to about 1 minutes, than we have to set the parameter about 10 and it will enable the feature only for long time massive reading (after ~100 seconds). So when we use short-reading and wanted all of them it the cache we will have it (except of evicted of course). When we use long-term heavy reading the featue will enabled after some time and birng better performance. *hbase.lru.cache.heavy.eviction.mb.size.limit* - set how many bytes desirable putting into BlockCache (and evicted from it). The feature will try to reach this value and maintan it. Don't try to set it too small because it lead to premature exit from this mode. For powerful CPU (about 20-40 physical cores) it could be about 400-500 MB. Average system (~10 cores) 200-300 MB. Some weak system (2-5 cores) maybe good with 50-100 MB. How it works: we set the limit and after each ~10 second caluclate how many bytes were freed. Overhead = Freed Bytes Sum (MB) * 100 / Limit (MB) - 100; For example we set the limit = 500 and were evicted 2000 MB. Overhead is: 2000 * 100 / 500 - 100 = 300% The feature is going to reduce a percent caching data blocks and fit evicted bytes closer to 100% (500 MB). So kind of an auto-scaling. If freed bytes less then the limit we have got negative overhead, for example if were freed 200 MB: 200 * 100 / 500 - 100 = -60% The feature will increase the percent of caching blocks and fit evicted bytes closer to 100% (500 MB). The current situation we can found in the log of RegionServer: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 < no eviction, 100% blocks is caching BlockCache evicted (MB): 2000, overhead (%): 300, heavy eviction counter: 1, current caching DataBlock (%): 97 < eviction begin, reduce of caching blocks It help to tune your system and find out what value is better set. Don't try to reach 0% overhead, it is impossible. Quite good 30-100% overhead, it prevent premature exit from this mode. *hbase.lru.cache.heavy.eviction.overhead.coefficient* - set how fast we want to get the result. If we know that our heavy reading for a long time, we don't want to wait and can increase the coefficient and get good performance sooner. But if we don't sure we can do it slowly and it could prevent premature exit from this mode. So, when the coefficient is higher we can get better performance when heavy reading is stable. But when reading is changing we can adjust to it and set the coefficient to lower value. For example, we set the coefficient = 0.01. It means the overhead (see above) will be multiplied by 0.01 and the result is value of reducing percent caching blocks. For example, if the overhead = 300% and the coefficient = 0.01, than percent of chaching blocks will reduce by 3%. Similar logic when overhead has got negative value (overshooting). Mayby it is just short-term fluctuation and we will ty to stay in this mode. It help avoid permature exit during short-term fluctuation. Backpressure has simple logic: more overshooting - more caching blocks. !schema.png! Finally, how to work reducing percent of caching blocks. Imagine we have very little cache, where can fit only 1 block and we are trying to read 3 blocks with offsets: 124 198 223 Without the feature, or when *hbase.lru.cache.heavy.eviction.count.limit* = 2147483647 we will put the block: 124, then put 198, evict 124, put 223, evict 198 A lot of work (5 actions and 2 evictions). With the feature and hbase.lru.cache.heavy.eviction.count.limit = 0 and the auto-scaling have reached to skip 97% of caching blocks (see the part of log above). The last few di
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127551#comment-17127551 ] Danil Lipovoy commented on HBASE-23887: --- Another one test - I wanted to see how to will work auto-scaling when we have changing load. So I run this scenario nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 5 -p fieldcount=1 -p operationcount=4 -s -t & sleep 100 nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 15 -p fieldcount=1 -p operationcount=6 -s -t & sleep 100 nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 5 -p fieldcount=1 -p operationcount=4 -s -t & sleep 100 nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 5 -p fieldcount=1 -p operationcount=4 -s -t & sleep 100 nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 20 -p fieldcount=1 -p operationcount=5 -s -t & sleep 100 nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 5 -p fieldcount=1 -p operationcount=2 -s -t & sleep 100 nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 5 -p fieldcount=1 -p operationcount=1 -s -t & sleep 100 nohup bin/ycsb run hbase2 -cp ~/hbase_conf -P workloads/select_u -p table=tbl4 -p columnfamily=cf -threads 10 -p fieldcount=1 -p operationcount=6 -s -t & with param: hbase.lru.cache.heavy.eviction.count.limit = 10 ( = disable the feature) Then I set: hbase.lru.cache.heavy.eviction.count.limit = 0 And have done almost the same scenario just set "sleep 50" because it works faster. The results: !wave.png! !image-2020-06-07-12-07-30-307.png! How it looks in the log: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction counter: 0, current caching DataBlock (%): 100 BlockCache evicted (MB): 5472, overhead (%): 2636, heavy eviction counter: 1, current caching DataBlock (%): 85 < test begin BlockCache evicted (MB): 6498, overhead (%): 3149, heavy eviction counter: 2, current caching DataBlock (%): 70 BlockCache evicted (MB): 5017, overhead (%): 2408, heavy eviction counter: 3, current caching DataBlock (%): 55 BlockCache evicted (MB): 3990, overhead (%): 1895, heavy eviction counter: 4, current caching DataBlock (%): 40 BlockCache evicted (MB): 2623, overhead (%): 1211, heavy eviction counter: 5, current caching DataBlock (%): 28 BlockCache evicted (MB): 2166, overhead (%): 983, heavy eviction counter: 6, current caching DataBlock (%): 19 BlockCache evicted (MB): 1254, overhead (%): 527, heavy eviction counter: 7, current caching DataBlock (%): 14 BlockCache evicted (MB): 456, overhead (%): 128, heavy eviction counter: 8, current caching DataBlock (%): 13 BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 9, current caching DataBlock (%): 13 BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 9, current caching DataBlock (%): 18 BlockCache evicted (MB): 456, overhead (%): 128, heavy eviction counter: 10, current caching DataBlock (%): 17 BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 11, current caching DataBlock (%): 17 BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 12, current caching DataBlock (%): 17 BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 13, current caching DataBlock (%): 17 BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 13, current caching DataBlock (%): 22 BlockCache evicted (MB): 798, overhead (%): 299, heavy eviction counter: 14, current caching DataBlock (%): 20 BlockCache evicted (MB): 684, overhead (%): 242, heavy eviction counter: 15, current caching DataBlock (%): 18 BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 16, current caching DataBlock (%): 17 BlockCache evicted (MB): 456, overhead (%): 128, heavy eviction counter: 17, current caching DataBlock (%): 16 BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 18, current caching DataBlock (%): 16 BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 19, current caching DataBlock (%): 16 BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 19, current caching DataBlock (%): 21 BlockCache evicted (MB): 684, overhead (%): 242, heavy eviction counter: 20, current caching DataBlock (%): 19 BlockCache evicted (MB): 456, overhea
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: image-2020-06-07-12-07-30-307.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, read_requests_100pBC_vs_23pBC.png,
> requests_100p.png, requests_100p.png, requests_new2_100p.png,
> requests_new_100p.png, scan.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: image-2020-06-07-12-07-24-903.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, read_requests_100pBC_vs_23pBC.png,
> requests_100p.png, requests_100p.png, requests_new2_100p.png,
> requests_new_100p.png, scan.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: wave.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, read_requests_100pBC_vs_23pBC.png,
> requests_100p.png, requests_100p.png, requests_new2_100p.png,
> requests_new_100p.png, scan.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127416#comment-17127416
]
Danil Lipovoy edited comment on HBASE-23887 at 6/7/20, 5:19 AM:
[~bharathv]
I have found there is no difference in performance while we are scanning:
!scan.png!
The cause looks like low GC during the scan. So it doesn't matter what kind of
BC to check. I think when we scan we got the bottle neck in another place (it
is not obvious where) and that's why results the same.
Another thing, I a little bit changed logic of calculation which of percent we
have to skip (cache.cacheDataBlockPercent).
I added the new param that help to control it:
{color:#067d17}hbase.lru.cache.heavy.eviction.overhead.coefficient =
0.01{color} {color:#172b4d}(heavyEvictionOverheadCoefficient){color}
{code:java}
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionMbSizeLimit) - 100;
...
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
int ch = (int) (freedDataOverheadPercent *
cache.heavyEvictionOverheadCoefficient);
ch = ch > 15 ? 15 : ch;
ch = ch < 0 ? 0 : ch;
cache.cacheDataBlockPercent -= ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent < 1 ? 1 :
cache.cacheDataBlockPercent;
}
{code}
And when we go below
*hbase.lru.cache.heavy.eviction.mb.size.limit*
{color:#172b4d}We use backward pressure:{color}
{code:java}
if (mbFreedSum >= cache.heavyEvictionMbSizeLimit * 0.1) {
// It help avoid exit during short-term fluctuation
int ch = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent > 100 ? 100 :
cache.cacheDataBlockPercent;
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
{code}
(full PR here: [https://github.com/apache/hbase/pull/1257/files])
{color:#172b4d}How it looks:{color}
!image-2020-06-07-08-19-00-922.png!
So, when GC works hard is reduce percent of cached blocks. When we jump below
the level, it help come back:
BlockCache evicted (MB): 4902, overhead (%): 2351, heavy eviction counter: 1,
current caching DataBlock (%): 85 < too much, fast slow down
BlockCache evicted (MB): 5700, overhead (%): 2750, heavy eviction counter: 2,
current caching DataBlock (%): 70
BlockCache evicted (MB): 5930, overhead (%): 2865, heavy eviction counter: 3,
current caching DataBlock (%): 55
BlockCache evicted (MB): 4446, overhead (%): 2123, heavy eviction counter: 4,
current caching DataBlock (%): 40
BlockCache evicted (MB): 3078, overhead (%): 1439, heavy eviction counter: 5,
current caching DataBlock (%): 26
BlockCache evicted (MB): 1710, overhead (%): 755, heavy eviction counter: 6,
current caching DataBlock (%): 19 < easy
BlockCache evicted (MB): 1026, overhead (%): 413, heavy eviction counter: 7,
current caching DataBlock (%): 15
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 8,
current caching DataBlock (%): 14 < easy
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 9,
current caching DataBlock (%): 14
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 10,
current caching DataBlock (%): 14
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 10,
current caching DataBlock (%): 19 < back pressure
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 11,
current caching DataBlock (%): 18
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 12,
current caching DataBlock (%): 17
BlockCache evicted (MB): 456, overhead (%): 128, heavy eviction counter: 13,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 14,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 15,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 16,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 17,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 18,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 19,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 20,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 21,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 22,
current caching DataBlock (%): 16
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 22,
current caching DataBlock (%): 21 < back pressure
BlockCache evicted (MB): 798, overhead (%): 299, h
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127416#comment-17127416
]
Danil Lipovoy edited comment on HBASE-23887 at 6/7/20, 5:11 AM:
[~bharathv]
I have found there is no difference in performance while we are scanning:
!scan.png!
The cause looks like low GC during the scan. So it doesn't matter what kind of
BC to check. I think when we scan we got the bottle neck in another place (it
is not obvious where) and that's why results the same.
Another thing, I a little bit changed logic of calculation which of percent we
have to skip (cache.cacheDataBlockPercent).
I added the new param that help to control it:
{color:#067d17}hbase.lru.cache.heavy.eviction.overhead.coefficient =
0.01{color} {color:#172b4d}(heavyEvictionOverheadCoefficient){color}
{code:java}
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionMbSizeLimit) - 100;
...
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
int ch = (int) (freedDataOverheadPercent *
cache.heavyEvictionOverheadCoefficient);
ch = ch > 15 ? 15 : ch;
ch = ch < 0 ? 0 : ch;
cache.cacheDataBlockPercent -= ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent < 1 ? 1 :
cache.cacheDataBlockPercent;
}
{code}
And when we go below
*hbase.lru.cache.heavy.eviction.mb.size.limit*
{color:#172b4d}We use backward pressure:{color}
{code:java}
if (mbFreedSum >= cache.heavyEvictionMbSizeLimit * 0.1) {
// It help avoid exit during short-term fluctuation
int ch = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent > 100 ? 100 :
cache.cacheDataBlockPercent;
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
{code}
(full PR here: [https://github.com/apache/hbase/pull/1257/files])
{color:#172b4d}How it looks:{color}
!image-2020-06-07-08-11-11-929.png!
So, when GC works hard is reduce percent of cached blocks. When we jump below
the level, it help come back:
BlockCache evicted (MB): 4902, overhead (%): 2351, heavy eviction counter: 1,
current caching DataBlock (%): 85 < too much, fast slow down
BlockCache evicted (MB): 5700, overhead (%): 2750, heavy eviction counter: 2,
current caching DataBlock (%): 70
BlockCache evicted (MB): 5930, overhead (%): 2865, heavy eviction counter: 3,
current caching DataBlock (%): 55
BlockCache evicted (MB): 4446, overhead (%): 2123, heavy eviction counter: 4,
current caching DataBlock (%): 40
BlockCache evicted (MB): 3078, overhead (%): 1439, heavy eviction counter: 5,
current caching DataBlock (%): 26
BlockCache evicted (MB): 1710, overhead (%): 755, heavy eviction counter: 6,
current caching DataBlock (%): 19 < easy
BlockCache evicted (MB): 1026, overhead (%): 413, heavy eviction counter: 7,
current caching DataBlock (%): 15
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 8,
current caching DataBlock (%): 14 < easy
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 9,
current caching DataBlock (%): 14
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 10,
current caching DataBlock (%): 14
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 10,
current caching DataBlock (%): 19 < back pressure
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 11,
current caching DataBlock (%): 18
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 12,
current caching DataBlock (%): 17
BlockCache evicted (MB): 456, overhead (%): 128, heavy eviction counter: 13,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 14,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 15,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 16,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 17,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 18,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 19,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 20,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 21,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 22,
current caching DataBlock (%): 16
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 22,
current caching DataBlock (%): 21 < back pressure
BlockCache evicted (MB): 798, overhead (%)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127416#comment-17127416
]
Danil Lipovoy edited comment on HBASE-23887 at 6/6/20, 6:42 PM:
[~bharathv]
I have found there is no difference in performance while we are scanning:
!scan.png!
The cause looks like low GC during the scan. So it doesn't matter what kind of
BC to check. I think when we scan we got the bottle neck in another place (it
is not obvious where) and that's why results the same.
Another thing, I a little bit changed logic of calculation which of percent we
have to skip (cache.cacheDataBlockPercent).
I added the new param that help to control it:
{color:#067d17}hbase.lru.cache.heavy.eviction.overhead.coefficient
{color:#172b4d}= 0.01{color}
{color:#172b4d}(heavyEvictionOverheadCoefficient){color}{color}
{code:java}
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionMbSizeLimit) - 100;
...
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
int ch = (int) (freedDataOverheadPercent *
cache.heavyEvictionOverheadCoefficient);
ch = ch > 15 ? 15 : ch;
ch = ch < 0 ? 0 : ch;
cache.cacheDataBlockPercent -= ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent < 1 ? 1 :
cache.cacheDataBlockPercent;
}
{code}
And when we go below
{color:#067d17}*hbase.lru.cache.heavy.eviction.mb.size.limit*{color}
{color:#172b4d}We use backward pressure:{color}
{code:java}
if (mbFreedSum >= cache.heavyEvictionMbSizeLimit * 0.1) {
// It help avoid exit during short-term fluctuation
int ch = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent > 100 ? 100 :
cache.cacheDataBlockPercent;
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
{code}
(full PR here: https://github.com/apache/hbase/pull/1257/files)
{color:#172b4d}How it looks:{color}
{color:#172b4d}!graph.png!{color}
So, when GC works hard is reduce percent of cached blocks. When we jump below
the level, it help come back:
BlockCache evicted (MB): 4902, overhead (%): 2351, heavy eviction counter: 1,
current caching DataBlock (%): 85 < too much, fast slow down
BlockCache evicted (MB): 5700, overhead (%): 2750, heavy eviction counter: 2,
current caching DataBlock (%): 70
BlockCache evicted (MB): 5930, overhead (%): 2865, heavy eviction counter: 3,
current caching DataBlock (%): 55
BlockCache evicted (MB): 4446, overhead (%): 2123, heavy eviction counter: 4,
current caching DataBlock (%): 40
BlockCache evicted (MB): 3078, overhead (%): 1439, heavy eviction counter: 5,
current caching DataBlock (%): 26
BlockCache evicted (MB): 1710, overhead (%): 755, heavy eviction counter: 6,
current caching DataBlock (%): 19 < easy
BlockCache evicted (MB): 1026, overhead (%): 413, heavy eviction counter: 7,
current caching DataBlock (%): 15
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 8,
current caching DataBlock (%): 14 < easy
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 9,
current caching DataBlock (%): 14
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 10,
current caching DataBlock (%): 14
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 10,
current caching DataBlock (%): 19 < back pressure
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 11,
current caching DataBlock (%): 18
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 12,
current caching DataBlock (%): 17
BlockCache evicted (MB): 456, overhead (%): 128, heavy eviction counter: 13,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 14,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 15,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 16,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 17,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 18,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 19,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 20,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 21,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 22,
current caching DataBlock (%): 16
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 22,
current caching DataBlock (%): 21 < back pressure
BlockCache e
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: requests_new2_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127416#comment-17127416
]
Danil Lipovoy commented on HBASE-23887:
---
[~bharathv]
I have found there is no difference in performance while we are scanning:
!scan.png!
The cause looks like low GC during the scan. So it doesn't matter what kind of
BC to check. I think when we scan we got the bottle neck in another place (it
is not obvious where) and that's why results the same.
Another thing, I a little bit changed logic of calculation which of percent we
have to skip (cache.cacheDataBlockPercent).
I added new param that help to control it
{color:#067d17}hbase.lru.cache.heavy.eviction.overhead.coefficient = 0.01
(heavyEvictionOverheadCoefficient)
{color}
{code:java}
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionMbSizeLimit) - 100;
...
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
int ch = (int) (freedDataOverheadPercent *
cache.heavyEvictionOverheadCoefficient);
ch = ch > 15 ? 15 : ch;
ch = ch < 0 ? 0 : ch;
cache.cacheDataBlockPercent -= ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent < 1 ? 1 :
cache.cacheDataBlockPercent;
}
{code}
And when we go below
{color:#067d17}*hbase.lru.cache.heavy.eviction.mb.size.limit*{color}
{color:#172b4d}We use backward pressure:
{color}
{code:java}
if (mbFreedSum >= cache.heavyEvictionMbSizeLimit * 0.1) {
// It help avoid exit during short-term fluctuation
int ch = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += ch;
cache.cacheDataBlockPercent = cache.cacheDataBlockPercent > 100 ? 100 :
cache.cacheDataBlockPercent;
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
{code}
{color:#172b4d}How it looks:{color}
{color:#172b4d}!graph.png!{color}
So, when GC works hard is reduce percent of cached blocks. When we jump below
the level, it help come back:
BlockCache evicted (MB): 4902, overhead (%): 2351, heavy eviction counter: 1,
current caching DataBlock (%): 85 < too much, fast slow down
BlockCache evicted (MB): 5700, overhead (%): 2750, heavy eviction counter: 2,
current caching DataBlock (%): 70
BlockCache evicted (MB): 5930, overhead (%): 2865, heavy eviction counter: 3,
current caching DataBlock (%): 55
BlockCache evicted (MB): 4446, overhead (%): 2123, heavy eviction counter: 4,
current caching DataBlock (%): 40
BlockCache evicted (MB): 3078, overhead (%): 1439, heavy eviction counter: 5,
current caching DataBlock (%): 26
BlockCache evicted (MB): 1710, overhead (%): 755, heavy eviction counter: 6,
current caching DataBlock (%): 19 < easy
BlockCache evicted (MB): 1026, overhead (%): 413, heavy eviction counter: 7,
current caching DataBlock (%): 15
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 8,
current caching DataBlock (%): 14 < easy
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 9,
current caching DataBlock (%): 14
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 10,
current caching DataBlock (%): 14
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 10,
current caching DataBlock (%): 19 < back pressure
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 11,
current caching DataBlock (%): 18
BlockCache evicted (MB): 570, overhead (%): 185, heavy eviction counter: 12,
current caching DataBlock (%): 17
BlockCache evicted (MB): 456, overhead (%): 128, heavy eviction counter: 13,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 14,
current caching DataBlock (%): 16
BlockCache evicted (MB): 342, overhead (%): 71, heavy eviction counter: 15,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 16,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 17,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 18,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 19,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 20,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 21,
current caching DataBlock (%): 16
BlockCache evicted (MB): 228, overhead (%): 14, heavy eviction counter: 22,
current caching DataBlock (%): 16
BlockCache evicted (MB): 114, overhead (%): -43, heavy eviction counter: 22,
current caching DataBlock (%): 21 < back pressure
BlockCache evicted (MB): 798, overhead (%): 299, heavy eviction counter: 23,
current caching DataBlock (%): 19
BlockCache evicted (MB): 684, overhead (%): 242, heavy eviction counter: 24,
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: graph.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> graph.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new_100p.png, scan.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: scan.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new_100p.png, scan.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[ https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123362#comment-17123362 ] Danil Lipovoy edited comment on HBASE-23887 at 6/2/20, 5:54 AM: Did more tests with the same tables, but in this time _recordcount_ = count of records in the table and *hbase.lru.cache.heavy.eviction.count.limit* = 0 *hbase.lru.cache.heavy.eviction.mb.size.limit* = 200 The results: !requests_new_100p.png! And YCSB stats: | |*original*|*feature*|*%*| |tbl1-u (ops/sec)|29,601|39,088|132| |tbl2-u (ops/sec)|38,793|61,692|159| |tbl3-u (ops/sec)|38,216|60,415|158| |tbl4-u (ops/sec)|325|657|202| |tbl1-z (ops/sec)|46,990|58,252|124| |tbl2-z (ops/sec)|54,401|72,484|133| |tbl3-z (ops/sec)|57,100|69,984|123| |tbl4-z (ops/sec)|452|763|169| |tbl1-l (ops/sec)|56,001|63,804|114| |tbl2-l (ops/sec)|68,700|76,074|111| |tbl3-l (ops/sec)|64,189|72,229|113| |tbl4-l (ops/sec)|619|897|145| | | | | | | | | | | | |*original*|*feature*|*%*| |tbl1-u AverageLatency(us)|1,686|1,276|76| |tbl2-u AverageLatency(us)|1,287|808|63| |tbl3-u AverageLatency(us)|1,306|825|63| |tbl4-u AverageLatency(us)|76,810|38,007|49| |tbl1-z AverageLatency(us)|1,061|856|81| |tbl2-z AverageLatency(us)|917|688|75| |tbl3-z AverageLatency(us)|873|712|82| |tbl4-z AverageLatency(us)|55,114|32,670|59| |tbl1-l AverageLatency(us)|890|781|88| |tbl2-l AverageLatency(us)|726|655|90| |tbl3-l AverageLatency(us)|777|690|89| |tbl4-l AverageLatency(us)|40,235|27,774|69| | | | | | | | | | | | |*original*|*feature*|*%*| |tbl1-u 95thPercentileLatency(us)|2,831|2,569|91| |tbl2-u 95thPercentileLatency(us)|1,266|1,073|85| |tbl3-u 95thPercentileLatency(us)|1,497|1,194|80| |tbl4-u 95thPercentileLatency(us)|370,943|49,471|13| |tbl1-z 95thPercentileLatency(us)|1,784|1,669|94| |tbl2-z 95thPercentileLatency(us)|918|871|95| |tbl3-z 95thPercentileLatency(us)|978|933|95| |tbl4-z 95thPercentileLatency(us)|336,639|48,863|15| |tbl1-l 95thPercentileLatency(us)|1,523|1,441|95| |tbl2-l 95thPercentileLatency(us)|820|825|101| |tbl3-l 95thPercentileLatency(us)|918|907|99| |tbl4-l 95thPercentileLatency(us)|77,951|48,575|62| was (Author: pustota): Did more tests with the same tables, but in this time _recordcount_ = count of records in the table and *hbase.lru.cache.heavy.eviction.count.limit* = 0 *hbase.lru.cache.heavy.eviction.mb.size.limit* = 200 The results: !requests_new_100p.png! sdYCSB stats: | |*original*|*feature*|*%*| |tbl1-u (ops/sec)|29,601|39,088|132| |tbl2-u (ops/sec)|38,793|61,692|159| |tbl3-u (ops/sec)|38,216|60,415|158| |tbl4-u (ops/sec)|325|657|202| |tbl1-z (ops/sec)|46,990|58,252|124| |tbl2-z (ops/sec)|54,401|72,484|133| |tbl3-z (ops/sec)|57,100|69,984|123| |tbl4-z (ops/sec)|452|763|169| |tbl1-l (ops/sec)|56,001|63,804|114| |tbl2-l (ops/sec)|68,700|76,074|111| |tbl3-l (ops/sec)|64,189|72,229|113| |tbl4-l (ops/sec)|619|897|145| | | | | | | | | | | | |*original*|*feature*|*%*| |tbl1-u AverageLatency(us)|1,686|1,276|76| |tbl2-u AverageLatency(us)|1,287|808|63| |tbl3-u AverageLatency(us)|1,306|825|63| |tbl4-u AverageLatency(us)|76,810|38,007|49| |tbl1-z AverageLatency(us)|1,061|856|81| |tbl2-z AverageLatency(us)|917|688|75| |tbl3-z AverageLatency(us)|873|712|82| |tbl4-z AverageLatency(us)|55,114|32,670|59| |tbl1-l AverageLatency(us)|890|781|88| |tbl2-l AverageLatency(us)|726|655|90| |tbl3-l AverageLatency(us)|777|690|89| |tbl4-l AverageLatency(us)|40,235|27,774|69| | | | | | | | | | | | |*original*|*feature*|*%*| |tbl1-u 95thPercentileLatency(us)|2,831|2,569|91| |tbl2-u 95thPercentileLatency(us)|1,266|1,073|85| |tbl3-u 95thPercentileLatency(us)|1,497|1,194|80| |tbl4-u 95thPercentileLatency(us)|370,943|49,471|13| |tbl1-z 95thPercentileLatency(us)|1,784|1,669|94| |tbl2-z 95thPercentileLatency(us)|918|871|95| |tbl3-z 95thPercentileLatency(us)|978|933|95| |tbl4-z 95thPercentileLatency(us)|336,639|48,863|15| |tbl1-l 95thPercentileLatency(us)|1,523|1,441|95| |tbl2-l 95thPercentileLatency(us)|820|825|101| |tbl3-l 95thPercentileLatency(us)|918|907|99| |tbl4-l 95thPercentileLatency(us)|77,951|48,575|62| > BlockCache performance improve by reduce eviction rate > -- > > Key: HBASE-23887 > URL: https://issues.apache.org/jira/browse/HBASE-23887 > Project: HBase > Issue Type: Improvement > Components: BlockCache, Performance >Reporter: Danil Lipovoy >Priority: Minor > Attachments: 1582787018434_rs_metrics.jpg, > 1582801838065_rs_metrics_new.png, BC_LongRun.png, > BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png, > eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png, > read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png, > requests_new_100p.png > > > Hi! > I first time here, correct me please if something wrong. > I want propose how to impro
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123362#comment-17123362
]
Danil Lipovoy commented on HBASE-23887:
---
Did more tests with the same tables, but in this time _recordcount_ = count of
records in the table and
*hbase.lru.cache.heavy.eviction.count.limit* = 0
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
The results:
!requests_new_100p.png!
sdYCSB stats:
| |*original*|*feature*|*%*|
|tbl1-u (ops/sec)|29,601|39,088|132|
|tbl2-u (ops/sec)|38,793|61,692|159|
|tbl3-u (ops/sec)|38,216|60,415|158|
|tbl4-u (ops/sec)|325|657|202|
|tbl1-z (ops/sec)|46,990|58,252|124|
|tbl2-z (ops/sec)|54,401|72,484|133|
|tbl3-z (ops/sec)|57,100|69,984|123|
|tbl4-z (ops/sec)|452|763|169|
|tbl1-l (ops/sec)|56,001|63,804|114|
|tbl2-l (ops/sec)|68,700|76,074|111|
|tbl3-l (ops/sec)|64,189|72,229|113|
|tbl4-l (ops/sec)|619|897|145|
| | | | |
| | | | |
| |*original*|*feature*|*%*|
|tbl1-u AverageLatency(us)|1,686|1,276|76|
|tbl2-u AverageLatency(us)|1,287|808|63|
|tbl3-u AverageLatency(us)|1,306|825|63|
|tbl4-u AverageLatency(us)|76,810|38,007|49|
|tbl1-z AverageLatency(us)|1,061|856|81|
|tbl2-z AverageLatency(us)|917|688|75|
|tbl3-z AverageLatency(us)|873|712|82|
|tbl4-z AverageLatency(us)|55,114|32,670|59|
|tbl1-l AverageLatency(us)|890|781|88|
|tbl2-l AverageLatency(us)|726|655|90|
|tbl3-l AverageLatency(us)|777|690|89|
|tbl4-l AverageLatency(us)|40,235|27,774|69|
| | | | |
| | | | |
| |*original*|*feature*|*%*|
|tbl1-u 95thPercentileLatency(us)|2,831|2,569|91|
|tbl2-u 95thPercentileLatency(us)|1,266|1,073|85|
|tbl3-u 95thPercentileLatency(us)|1,497|1,194|80|
|tbl4-u 95thPercentileLatency(us)|370,943|49,471|13|
|tbl1-z 95thPercentileLatency(us)|1,784|1,669|94|
|tbl2-z 95thPercentileLatency(us)|918|871|95|
|tbl3-z 95thPercentileLatency(us)|978|933|95|
|tbl4-z 95thPercentileLatency(us)|336,639|48,863|15|
|tbl1-l 95thPercentileLatency(us)|1,523|1,441|95|
|tbl2-l 95thPercentileLatency(us)|820|825|101|
|tbl3-l 95thPercentileLatency(us)|918|907|99|
|tbl4-l 95thPercentileLatency(us)|77,951|48,575|62|
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many t
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: requests_new_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 9:18 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of auto-scaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation. Will be fine add more logic here.
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_requestdistribution=uniform_
Workload scenario "z":
_requestdistribution=zipfian_
Workload scenario "l":
_requestdistribution=latest_
Other parameters:
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_recordcount=100 (I just noticed this value is too small, I will provide
new tests with bigger value later)_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature:
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reached.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120565#comment-17120565
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 8:44 PM:
-
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the algorithm, that I posted above (will
add changes to PR quite soon). It is good when numbers of reading requests are
changing. Looks like the new approach copes well with wide variety kind of
situation (a lot of tests in the next messages after answers).
1. I'm nor sure, but maybe it is because first few seconds, while BlockCache
is empty, my old version of realization prevented effective populating the BC.
I mean it was skipping blocks when eviction is not running - and a lot of
blocks could be cached but were lost. With the new approach the problems has
gone. For example:
This is when 100% of data caching (uniform distribution):
[OVERALL], RunTime(ms), 1506417
[OVERALL], Throughput(ops/sec), 33191.34077748724
[TOTAL_GCS_PS_Scavenge], Count, 8388
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 12146
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8062840501667201
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 22
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0014604189942094387
[TOTAL_GCs], Count, 8389
[TOTAL_GC_TIME], Time(ms), 12168
[TOTAL_GC_TIME_%], Time(%), 0.8077444691609296
[READ], Operations, 5000
[READ], AverageLatency(us), 1503.45024378
[READ], MinLatency(us), 137
[READ], MaxLatency(us), 383999
[READ], 95thPercentileLatency(us), 2231
[READ], 99thPercentileLatency(us), 13503
[READ], Return=OK, 5000
The same table with the patch:
[OVERALL], RunTime(ms), 1073257
[OVERALL], Throughput(ops/sec), 46587.1641181935
[TOTAL_GCS_PS_Scavenge], Count, 7201
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 9799
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9130152423883563
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 23
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.002143009549436901
[TOTAL_GCs], Count, 7202
[TOTAL_GC_TIME], Time(ms), 9822
[TOTAL_GC_TIME_%], Time(%), 0.9151582519377931
[READ], Operations, 5000
[READ], AverageLatency(us), 1070.52889804
[READ], MinLatency(us), 142
[READ], MaxLatency(us), 327167
[READ], 95thPercentileLatency(us), 2071
[READ], 99thPercentileLatency(us), 6539
[READ], Return=OK, 5000
The same picture show all other test - you could see details below.
2.Looks like it could make negative effect if we try to use the feature if we
set *hbase.lru.cache.heavy.eviction.count.limit*=0 and
*hbase.lru.cache.heavy.eviction.mb.size.limit*=1 and doing sporadly short
reading the same data. I meant when size BC=3 and we read block 1,2,3,4,3,4 ...
4,3,2,1,2,1 ... 1,2,3,4,3,4... In this scenario better save all blocks. But
this parameters will skip blocks which we will need quite soon. My opinion - it
is extremely good for massive long-term reading on powerful servers. For short
reading small amount of date too small values of the parameters could be
pathological.
3. If I understand you correct - you meant that after compaction real blocks
offset changed. But when HFiles compacted anyway all blocks removed from BC too.
4.Now we have two parameters for tuning:
*hbase.lru.cache.heavy.eviction.count.limit* - it controls how soon we want to
see eviction rate reduce. If we know that our load pattern is only long term
reading, we can set it 0. It means if we are reading - it is for a long time.
But if we have some times short reading the same data and some times long-term
reading - we have to divide it by this parameter. For example we know - our
short reading used to about 1 min, we have to set the param about 10 and it
will enable the feature only for long time massive reading.
*hbase.lru.cache.heavy.eviction.mb.size.limit* - it lets to control when we
sure that GC will be suffer. For weak PC it could be about 50-100 MB. For
powerful servers 300-500 MB.
I added some useful information into logging:
{color:#871094}LOG{color}.info({color:#067d17}"BlockCache evicted (MB): {},
overhead (%) {}, " {color}+
{color:#067d17}"heavy eviction counter {}, " {color}+
{color:#067d17}"current caching DataBlock (%): {}"{color},
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount,
{color:#00}cache{color}.{color:#871094}cacheDataBlockPercent{color});
It will help to understand what kind of values we have and how to tune it.
5. I think it is pretty good idea. Give me time, please, to do tests and check
what will be.
Well, I will post information about the tests in the next message.
was (Author: pustota):
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done t
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 8:43 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of auto-scaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation. Will be fine add more logic here.
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature:
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reached.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching Dat
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120565#comment-17120565
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 8:06 PM:
-
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the algorithm, that I posted above (will
add changes to PR quite soon). It is good when numbers of reading requests are
changing. Looks like the new approach copes well with wide variety kind of
situation (a lot of tests in the next messages after answers).
1. I'm nor sure, but maybe it is because first few seconds, while BlockCache
is empty, my old version of realization prevented effective populating the BC.
I mean it was skipping blocks when eviction is not running - and a lot of
blocks could be cached but were lost. With the new approach the problems has
gone. For example:
This is when 100% of data caching (uniform distribution):
[OVERALL], RunTime(ms), 1506417
[OVERALL], Throughput(ops/sec), 33191.34077748724
[TOTAL_GCS_PS_Scavenge], Count, 8388
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 12146
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8062840501667201
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 22
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0014604189942094387
[TOTAL_GCs], Count, 8389
[TOTAL_GC_TIME], Time(ms), 12168
[TOTAL_GC_TIME_%], Time(%), 0.8077444691609296
[READ], Operations, 5000
[READ], AverageLatency(us), 1503.45024378
[READ], MinLatency(us), 137
[READ], MaxLatency(us), 383999
[READ], 95thPercentileLatency(us), 2231
[READ], 99thPercentileLatency(us), 13503
[READ], Return=OK, 5000
The same table with the patch:
[OVERALL], RunTime(ms), 1073257
[OVERALL], Throughput(ops/sec), 46587.1641181935
[TOTAL_GCS_PS_Scavenge], Count, 7201
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 9799
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9130152423883563
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 23
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.002143009549436901
[TOTAL_GCs], Count, 7202
[TOTAL_GC_TIME], Time(ms), 9822
[TOTAL_GC_TIME_%], Time(%), 0.9151582519377931
[READ], Operations, 5000
[READ], AverageLatency(us), 1070.52889804
[READ], MinLatency(us), 142
[READ], MaxLatency(us), 327167
[READ], 95thPercentileLatency(us), 2071
[READ], 99thPercentileLatency(us), 6539
[READ], Return=OK, 5000
The same picture all other test - you could see details below.
2.Looks like it could make negative effect if we try to use the feature if we
set *hbase.lru.cache.heavy.eviction.count.limit*=0 and
*hbase.lru.cache.heavy.eviction.mb.size.limit*=1 and doing sporadly short
reading the same data. I meant when size BC=3 and we read block 1,2,3,4,3,4 ...
4,3,2,1,2,1 ... 1,2,3,4,3,4... In this scenario better save all blocks. But
this parameters will skip blocks which we will need quite soon. My opinion - it
is extremely good for massive long-term reading on powerful servers. For short
reading small amount of date too small values of the parameters could be
pathological.
3. If I understand you correct - you meant that after compaction real blocks
offset changed. But when HFiles compacted anyway all blocks removed from BC too.
4.Now we have two parameters for tuning:
*hbase.lru.cache.heavy.eviction.count.limit* - it controls how soon we want to
see eviction rate reduce. If we know that our load pattern is only long term
reading, we can set it 0. It means if we are reading - it is for a long time.
But if we have some times short reading the same data and some times long-term
reading - we have to divide it by this parameter. For example we know - our
short reading used to about 1 min, we have to set the param about 10 and it
will enable the feature only for long time massive reading.
*hbase.lru.cache.heavy.eviction.mb.size.limit* - it lets to control when we
sure that GC will be suffer. For weak CPU it could be about 50-100 MB. For
powerful servers 300-500 MB.
I added some useful information into logging:
{color:#871094}LOG{color}.info({color:#067d17}"BlockCache evicted (MB): {},
overhead (%) {}, " {color}+
{color:#067d17}"heavy eviction counter {}, " {color}+
{color:#067d17}"current caching DataBlock (%): {}"{color},
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount,
{color:#00}cache{color}.{color:#871094}cacheDataBlockPercent{color});
It will help to understand what kind of values we have and how to tune it.
4. I think it is pretty good idea. Give me time, please, to do tests and check
what will be.
Well, I will post information about the tests in the next message.
was (Author: pustota):
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the a
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 5:27 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation. Will be fine add more logic here.
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching Dat
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 5:05 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: Blo
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 5:01 PM:
-
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 4:56 PM:
-
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 4:55 PM:
-
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): 0, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): 0, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): 0, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761,
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120591#comment-17120591
]
Danil Lipovoy commented on HBASE-23887:
---
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 <- no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 <- start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 <- *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761, heavy
eviction counter: 7, current caching DataBlock (%):
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: requests_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Comment: was deleted
(was: All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor
(3150 MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 <- no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 <- start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 <- *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761, heavy
eviction counter: 7, current caching DataBlock (%): 88
LruBlockCache: BlockCa
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120590#comment-17120590
]
Danil Lipovoy commented on HBASE-23887:
---
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe
[here|https://issues.apache.org/jira/browse/HBASE-23887?focusedCommentId=17110503&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17110503]):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 <- no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 <- start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 <- *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: eviction_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: gc_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
