[jira] [Commented] (HBASE-22324) loss a mass of data when the sequenceId of cells greater than Integer.Max, because MemStoreMergerSegmentsIterator can not merge segments
[
https://issues.apache.org/jira/browse/HBASE-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833704#comment-16833704
]
Eshcar Hillel commented on HBASE-22324:
---
nice work.
nitpicking: the two test methods are not 100% the same one uses "this" in the
c'tor and the other does not.
Other than that +1.
Thanks again.
> loss a mass of data when the sequenceId of cells greater than Integer.Max,
> because MemStoreMergerSegmentsIterator can not merge segments
> --
>
> Key: HBASE-22324
> URL: https://issues.apache.org/jira/browse/HBASE-22324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: chenyang
>Priority: Blocker
> Labels: patch
> Fix For: 2.1.0
>
> Attachments: HBASE-22324.branch-2.1.0001.patch
>
>
> if your memstore type is CompactingMemStore，MemStoreMergerSegmentsIterator
> can not merge memstore segments when the seqId of cells greater than
> Integer.Max, as a result, lossing a mass of data. the reason is that
> MemStoreMergerSegmentsIterator use Integer.Max as readPt when create Scanner,
> but the seqId of cell may be greater than Integer.MAX_VALUE, it`s type is
> long. code as below:
> {code:java}
> public MemStoreMergerSegmentsIterator(List segments,
> CellComparator comparator,
> int compactionKVMax) throws IOException {
> super(compactionKVMax);
> // create the list of scanners to traverse over all the data
> // no dirty reads here as these are immutable segments
> AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners);
> //bug, should use Long.MAX_VALUE
> heap = new KeyValueHeap(scanners, comparator);
> }
> SegmentScanner.java code as below
> protected void updateCurrent() {
> Cell startKV = current;
> Cell next = null;
> try {
> while (iter.hasNext()) {
> next = iter.next();
> // here, if seqId>readPoint(Integer.MAX_VALUE), never read cell, as a
> result, lossing lots of cells
> if (next.getSequenceId() <= this.readPoint) {
> current = next;
> return;// skip irrelevant versions
> }
> if (stopSkippingKVsIfNextRow && // for backwardSeek() stay in the
> startKV != null &&// boundaries of a single row
> segment.compareRows(next, startKV) > 0) {
> current = null;
> return;
> }
> } // end of while
> current = null; // nothing found
> } finally {
> if (next != null) {
> // in all cases, remember the last KV we iterated to, needed for
> reseek()
> last = next;
> }
> }
> }
> MemStoreCompactorSegmentsIterator has the same bug
> public MemStoreCompactorSegmentsIterator(List segments,
> CellComparator comparator, int compactionKVMax, HStore store) throws
> IOException {
> super(compactionKVMax);
> List scanners = new ArrayList();
> AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners);
> //bug, should use Long.MAX_VALUE
> // build the scanner based on Query Matcher
> // reinitialize the compacting scanner for each instance of iterator
> compactingScanner = createScanner(store, scanners);
> refillKVS();
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22324) loss a mass of data when the sequenceId of cells greater than Integer.Max, because MemStoreMergerSegmentsIterator can not merge segments
[
https://issues.apache.org/jira/browse/HBASE-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831425#comment-16831425
]
Eshcar Hillel commented on HBASE-22324:
---
Thanks [~HB-CY].
Note that MemStoreCompactorSegmentsIterator has the same problem.
Worth to add a test to cover this bug.
> loss a mass of data when the sequenceId of cells greater than Integer.Max,
> because MemStoreMergerSegmentsIterator can not merge segments
> --
>
> Key: HBASE-22324
> URL: https://issues.apache.org/jira/browse/HBASE-22324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: chenyang
>Priority: Blocker
> Labels: patch
> Fix For: 2.1.0
>
> Attachments: HBASE-22324.branch-2.1.001.patch,
> HBASE-22324.branch-2.1.001.patch
>
>
> if your memstore type is CompactingMemStore，MemStoreMergerSegmentsIterator
> can not merge memstore segments when the seqId of cells greater than
> Integer.Max, as a result, lossing a mass of data. the reason is that
> MemStoreMergerSegmentsIterator use Integer.Max as readPt when create Scanner,
> but the seqId of cell may be greater than Integer.MAX_VALUE, it`s type is
> long. code as below:
> {code:java}
> public MemStoreMergerSegmentsIterator(List segments,
> CellComparator comparator,
> int compactionKVMax) throws IOException {
> super(compactionKVMax);
> // create the list of scanners to traverse over all the data
> // no dirty reads here as these are immutable segments
> AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners);
> //bug, should use Long.MAX_VALUE
> heap = new KeyValueHeap(scanners, comparator);
> }
> SegmentScanner.java code as below
> protected void updateCurrent() {
> Cell startKV = current;
> Cell next = null;
> try {
> while (iter.hasNext()) {
> next = iter.next();
> // here, if seqId>readPoint(Integer.MAX_VALUE), never read cell, as a
> result, lossing lots of cells
> if (next.getSequenceId() <= this.readPoint) {
> current = next;
> return;// skip irrelevant versions
> }
> if (stopSkippingKVsIfNextRow && // for backwardSeek() stay in the
> startKV != null &&// boundaries of a single row
> segment.compareRows(next, startKV) > 0) {
> current = null;
> return;
> }
> } // end of while
> current = null; // nothing found
> } finally {
> if (next != null) {
> // in all cases, remember the last KV we iterated to, needed for
> reseek()
> last = next;
> }
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820448#comment-16820448 ] Eshcar Hillel commented on HBASE-15560: --- Congratulations!! Great to see this finally get in :) > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542-addendum.master.005.patch, > HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, > HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, > HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683776#comment-16683776 ] Eshcar Hillel commented on HBASE-20542: --- It can work. But are you sure MOB should be treated with in-memory compaction? The recommendation for MOB [is to be stored directly in HDFS|https://www.cloudera.com/documentation/enterprise/5-4-x/topics/admin_hbase_mob.html] If MOB are the normal case then each time you insert a cell this will trigger in-memory flush, and after a few flushes -- in-memory compaction. This is wasteful in terms of resources utilizations. Have you considered using non-compacting memstore? > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542-addendum.master.005.patch, > HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, > HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, > HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683492#comment-16683492 ] Eshcar Hillel commented on HBASE-20542: --- bq. if cellSize larger than inmemoryFlushSize, AbstractMemStore#doAddOrUpsert will never break the while loop? In-memory compaction is aimed for columns with small size cells, and we should not see cells that are bigger than in-memory flush size. However, your claim is correct: If cellSize > inmemoryFlushSize, CompactingMemStore#shouldFlushInMemory breaks the while loop and returns true as the size is above flush threshold. In turn CompactingMemStore#checkAndAddToActiveSize flushes active segment into compaction pipeline, dispatches in-memory compaction thread, and returns false. This results in CompactingMemStore#preUpdate releasing the lock on the active segment and returning false, potentially hitting an infinite loop. We may want to throw an exception here instead of looping over and over again. One way for the application to deal with this issue is to set larger portion for active segments, so that cell size is always smaller than inmemoryFlushSize. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542-addendum.master.005.patch, > HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, > HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, > HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563885#comment-16563885 ] Eshcar Hillel commented on HBASE-20188: --- Hi, [~stack] I noticed you posted some additional profiling results from July in the above link. I was wondering, is there a timeline toward the 2.2.0 release? What are the trajectories? Also, are you planing to run YCSB performance benchmarks again? Would it be load A, run A run C? Any other benchmarks you would consider? We hope you will be able to benchmark IMC memstore and to consider making it the default. Last results we presented following HBASE-20542 show that IMC read performance is comparable to no-compaction, and is much better in the write-only workloads x. Please let us know if there is anything else we can do to help drive this effort. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Blocker > Fix For: 3.0.0, 2.2.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539724#comment-16539724 ] Eshcar Hillel commented on HBASE-20542: --- Pushed to master and branch-2 > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542-addendum.master.005.patch, > HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, > HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, > HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538313#comment-16538313 ] Eshcar Hillel commented on HBASE-20542: --- TestHStore overrides the internal implementation of CompactingMemStore to manipulate the timing of the compaction and count the number of times it is successfully invoked. Since the internals of CompactingMemStore were changed in this issue the test as well as the handles inside CompactingMemStore needed to be changed to allow the correct manipulation. TestCompactingToCellFlatMapMemStore was also affected by the changes in internal mechanism, specifically the fact that in memory flush is now happening as part of a write operation, and only the compaction runs in the background. The fix also helps here. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542-addendum.master.005.patch, > HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, > HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, > HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536901#comment-16536901 ] Eshcar Hillel commented on HBASE-20542: --- I will fix the style error and will commit the addendum to master and branch-2. It should fix the failing test. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542-addendum.master.005.patch, > HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, > HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, > HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542-addendum.master.005.patch > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542-addendum.master.005.patch, > HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, > HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, > HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Status: Patch Available (was: Reopened) > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, > HBASE-20542.master.005-addendum.patch, run.sh, workloada, workloadc, > workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542.master.005-addendum.patch > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, > HBASE-20542.master.005-addendum.patch, run.sh, workloada, workloadc, > workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532354#comment-16532354 ] Eshcar Hillel commented on HBASE-20542: --- No the threshold is already set in TestHStore. It failed for a different reason. Currently I cannot reproduce the failure. Lets see if it shows up in the flaky tests dashboard. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531187#comment-16531187 ] Eshcar Hillel commented on HBASE-20542: --- Thanks [~Apache9] for bringing this to my attention. The problem is simpler - memstore flush size is set to 64KB in these tests while in-memory flush is now only at 2MB by default; no in-memory flush means no in-memory compaction hence flush to disk. Fix is simple: setting in-memory flush size in this test to be 0.014x as was before. I will push an addendum. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529879#comment-16529879 ] Eshcar Hillel commented on HBASE-20542: --- OK then, closing this issue. Will follow on performance testing progress and 2.2 release in HBASE-20188 and in other issues if/when they come up. Thanks all for reviews and comments. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529187#comment-16529187 ] Eshcar Hillel commented on HBASE-20542: --- Thanks [~Apache9], pushed the patch to master and branch-2. So what are the plans for testing performance of 2.1.0 and ensuing releases? Is it all in the context of HBASE-20188? Running load (a) and then workloada and workloadc of YCSB? Would you also consider workloadx? The reason I ask is that we would like to suggest setting IMC as default again. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529035#comment-16529035 ] Eshcar Hillel commented on HBASE-20542: --- Thanks [~mdrob]. So except for these warnings, QA passed. I am pushing this patch to master. [~Apache9] [~stack] should I commit this to branch-2/branch2.0 or would you like to take a closer look at it? > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542.branch-2.005.patch > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542.master.003.patch > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.master.003.patch, run.sh, workloada, workloadc, workloadx, > workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524817#comment-16524817 ] Eshcar Hillel commented on HBASE-20542: --- I don't understand why I get these java warnings, they are related to a test that has nothing to do with this issue, and was not changed by this patch !?! > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542.branch-2.004.patch > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542.branch-2.003.patch > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, run.sh, workloada, workloadc, workloadx, > workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517160#comment-16517160 ] Eshcar Hillel commented on HBASE-20542: --- bq. Is it possible to see the patch on the review board? Thanks! I was having some trouble before, but now it is available. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, run.sh, workloada, > workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516256#comment-16516256 ] Eshcar Hillel commented on HBASE-20188: --- Please note new patch and benchmarks results are available in HBASE-20542 > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516253#comment-16516253 ] Eshcar Hillel commented on HBASE-20542: --- bq. Where is write lock released ? The write lock is not released. Once the write lock is acquired the segment is really immutable, no update operation should read-lock it and there is no need to release the write lock. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, run.sh, workloada, > workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516187#comment-16516187 ] Eshcar Hillel commented on HBASE-20542: --- I'm having some trouble with RB will check this and see how I can upload the patch there > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, run.sh, workloada, > workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516186#comment-16516186 ] Eshcar Hillel commented on HBASE-20542: --- Attaching the ycsb scripts used for benchmarking. Two sets of runs [^run.sh] . First, write-only zipfian [^workloadx] with 10 region pre-split, followed by a read-only zipfian [^workloady] reading only one column. Second is the standard uniform load (a), mixed read-write [^workloada] , read-only [^workloadc] reading all columns. This is a comparison of the average throughput and lift vs no-IMC: ||comp || index || workloadx || workloady || load || workloada || workloadc || | NONE | - | 49,369 | 17,682 | 11,010 |10,468 |7,779 | | BASIC | CAM | 57965 | 17,132 |11,854 | 10,318 | 7,552 | | | | +17.41% | -3.11% | +7.67%| -1.44%| -2.91% | |BASIC| CCM | 52,296 | 16,644 | 12,140 | 9,705 | 7,465 | | | | +6% | -6% | +10%| -7% | -4%| > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, run.sh, workloada, > workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: run.sh workloadx workloady workloadc workloada > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, run.sh, workloada, > workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516156#comment-16516156 ] Eshcar Hillel commented on HBASE-20542: --- Patch is attached. To reduce internal fragmentation the size of the active segment is set to be the size of one MSLAB chunk (by default 2MB). An add operation is supplemented with pre-update and post update procedures. The pre-update procedure atomically increases the size of the segment if this increment does not exceed the segment size threshold, and then continues with the normal path of updating the memstore. If the increment will exceed the segment size threshold then the size is not increased and instead (1) the segment is flushed into the compaction pipeline, (2) a new active segment is created, (3) an IMC task is scheduled in the background, (4) the operation re-runs the pre-update procedure, this time with the new active segment. This changes calls for an additional optimization. The IMC no longer needs to acquire the region level updates lock. Instead we use segment level read-write lock to synchronize IMC with concurrent update operations. This is better since with the new solution IMC only needs to wait only for those few operations that already updated the size of the segment in the pre-update procedure but are still updating the segment skip list, and does not need to wait for operations of other stores. Moreover, update operation do not wait for in-memory flush to complete as before. To synchronize, update operation take the read lock of the segment they are updating in the pre-update procedure, and release it in the post-update procedure. IMC thread take the write lock of each segment it is compacting. This ensures all updates that started before the in-memory flush have completed. I will upload the patch also in RB. Feel free to ask questions and comment. > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542.branch-2.001.patch Status: Patch Available (was: Open) > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20542.branch-2.001.patch > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499781#comment-16499781
]
Eshcar Hillel commented on HBASE-20628:
---
Patch looks good to me. Performance improvement is also impressive.
If the close boolean is just an optimization then it can definitely go in a
follow-on, I just wasn't sure if it was needed for correctness.
What about the scanner order? I noticed there were some discussions recently -
what was the conclusion: it is not necessary for in-memory segments since cells
have sequence id in-memory?
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001 (1).patch,
> HBASE-20628.branch-2.0.001.patch, HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.002.patch, HBASE-20628.branch-2.0.003.patch,
> HBASE-20628.branch-2.0.004.patch, Screen Shot 2018-05-25 at 9.38.00 AM.png,
> hits-20628.png, hits.003.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return Collections.singletonList(new
> CollectionBackedScanner(this.cellSet.get(), comparator));
> + }
> +
>/**
> * @return whether the segment has any cells
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Resolution: Fixed Status: Resolved (was: Patch Available) > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496218#comment-16496218 ] Eshcar Hillel commented on HBASE-20390: --- Great, so I'm closing this issue. Thanks all > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494848#comment-16494848 ] Eshcar Hillel commented on HBASE-20390: --- Documentation - I agree this is more confusing than helping. I will remove the mention of the chunk - it is more an internal implementation detail. In addition, we are working in HBASE-20542 to remove dependence upon this parameter. > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493739#comment-16493739 ] Eshcar Hillel commented on HBASE-20390: --- This is still the report following the first addendum, waiting for the report after the second addendum > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493527#comment-16493527 ] Eshcar Hillel commented on HBASE-20390: --- Pushed. Lets wait for Hudson reports. > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[
https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493518#comment-16493518
]
Eshcar Hillel commented on HBASE-20390:
---
:(:( Still OOME!! It worked on my machine
And anyway I see the threshold is even lower than what I computed, perhaps the
heap is even smaller than 1GB
{{in-memory flush size threshold=183.50 KB}}
I am upping the threshold to the maximum in which we still see in-memory flush
occurring. Hope this is the last change.
> IMC Default Parameters for 2.0.0
>
>
> Key: HBASE-20390
> URL: https://issues.apache.org/jira/browse/HBASE-20390
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Attachments: HBASE-20390-branch-2.0-01.patch,
> HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch,
> HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation -
> throughput SSD_HDD.pdf, hits.ihc.png
>
>
> Setting new default parameters for in-memory compaction based on performance
> tests done in HBASE-20188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493360#comment-16493360
]
Eshcar Hillel commented on HBASE-20628:
---
Thanks [~anoop.hbase] for the explanation. Now I get it.
So actually there is no reason or intention to seek for a key that was already
read. This is good.
Actually when reading the documentation of the reseek(cell) method this is what
is written there:
bq. Reseek the scanner at or after the specified KeyValue. This method is
guaranteed to seek at or after the required key *only if the key comes after
the current position of the scanner*. Should not be used to seek to a key which
may come before the current position.
So in this respect the implementation of CollectionBackedScanner::reseek(cell)
is correct, while the implementation of SegmentScanner::reseek(cell) seems to
be inefficient with no real reason.
I checked, no one changed this piece of code since I created this file 2 years
ago, and I simply copy-pasted it from another file
(DefaultMemStore/MemStoreScanner or else), which means this existed also in
hbase-1.4 and hence CollectionBackedScanner allowed for improved performance.
Here is a suggestion: replace the implementation of
SegmentScanner::reseek(cell) with the code from
CollectionBackedScanner::reseek(cell) which does not create a new iterator and
simply runs next until reaching the cell.
We can then in parallel (1) run regression tests to make sure we didn't violate
correctness, and (2) run the same benchmark to see if we get the same
performance boost.
If we get (/) in both fronts then I think it is better to keep just the
SegmentScanner and not use a special case scanner.
Reasonable?
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001 (1).patch,
> HBASE-20628.branch-2.0.001.patch, HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.002.patch, Screen Shot 2018-05-25 at 9.38.00 AM.png,
> hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493319#comment-16493319 ] Eshcar Hillel commented on HBASE-20390: --- Yesterday I pushed the one-liner to master and was waiting for the hudson report to make sure there are no additional surprises. But I won't wait longer and will just push it to branch-2 and branch-2.0 now. No flaky tests dashboard for master? ;) > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492572#comment-16492572
]
Eshcar Hillel commented on HBASE-20628:
---
We need to understand why and how reseek is used during snapshot scans. This is
following the quote from Anoop from back-channel communication
{quote}Seems the SegmentScanner is making the cost. Specially impl like
reseek(). We will get calls to reseek() so many times within a flush op...
{quote}
If we cannot fix the current SegmentScanner implementation to work efficiently
during a flush then yes I suggest to follow the patch in the description plus
implement {{getScannersForSnapshot()}} in {{CompositeImmutableSegment}}.
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001 (1).patch,
> HBASE-20628.branch-2.0.001.patch, HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.002.patch, Screen Shot 2018-05-25 at 9.38.00 AM.png,
> hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return Collections.singletonList(new
> CollectionBackedScanner(this.cellSet.get(), comparator));
> + }
> +
>/**
> * @return whether the segment has any cells
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492139#comment-16492139
]
Eshcar Hillel commented on HBASE-20628:
---
[~stack] - you probably missed my comment here last week, copy-pasting it here
again.
Also had one comment in RB.
Thanks.
Obviously any change that improves the performance without affecting
correctness is welcomed.
It is interesting though to understand what brings this improvement.
I don't think it is related to the fact that SegmentScanner supports backward
scans because this code is simply not utilised in the case of flush.
The method reseek(cell) is indeed very different in the two implementations:
SegmentScanner creates a new iterator which is done like this: {{return
segment.tailSet(cell).iterator();}} (could this be done in a more efficient
way?)
while CollectionBackedScanner is simply continuing with the same existing
iterator
Which is the correct behaviour? could it be that we ask to reseek for a cell
that precedes the current cell during a flush? Why even call reseek(cell) and
not only next() during a flush?
In addition, the current patch only solves the performance issue for the
Segment class (in the no-IMC implementation). What about solving this also for
CompositeImmutableSegment? If we follow the code in the description and not in
the patch it should be an easy fix.
Regarding the performance results - do you see the same affect when running
with --writeToWAL=true?
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001 (1).patch,
> HBASE-20628.branch-2.0.001.patch, HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.002.patch, Screen Shot 2018-05-25 at 9.38.00 AM.png,
> hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
>
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492036#comment-16492036 ] Eshcar Hillel commented on HBASE-20390: --- Pushing the patch to branches resulted in an out-of-memory exception in one of the tests that was not triggered in QA. Debugging the test, I found out this is a classic example of the opposite case of memory underutilisation. Make a long story short: in test heap size is 1GB from which 400MB allocated to memstore (defaults); 10 column families share this get 40MB each. At 0.014 in-memory flush factor store flushes a segment every ~500KB (25% utilisation of the mslab chunk). Eventually this causes the OOME. HBASE-20542 aims to protect against this kind of underutilisation as well. Simply setting the factor to 0.02 in this test triggers an in-memory flush every 750KB which apparently is sufficient for not causing OOME. Flush to disk are also frequent in this test so there is no point in setting a higher threshold - it will cause flushes to disk to happen before a flush in-memory happens. I will push a 1 line addendum to fix the test. > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488979#comment-16488979 ] Eshcar Hillel commented on HBASE-20390: --- Thanks [~chia7712] I updated src/main/asciidoc/_chapters/inmemory_compaction.adoc > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel reassigned HBASE-20542: - Assignee: Eshcar Hillel > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488062#comment-16488062 ] Eshcar Hillel commented on HBASE-20390: --- QA passed successfully. Recall that these parameters showed performance improvement wrt the previous default params, and we plan to continue the work to reduce internal fragmentation in HBASE-20542. I plan to commit this to master. Should I commit this to branch-2.1.0 [~Apache9]? Should I commit this to branch-2.0.1 [~chia7712]? Should I commit this to branch-2.0 [~stack]? > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488044#comment-16488044
]
Eshcar Hillel commented on HBASE-20628:
---
Obviously any change that improves the performance without affecting
correctness is welcomed.
It is interesting though to understand what brings this improvement.
I don't think it is related to the fact that SegmentScanner supports backward
scans because this code is simply not utilised in the case of flush.
The method reseek(cell) is indeed very different in the two implementations:
SegmentScanner creates a new iterator which is done like this: {{return
segment.tailSet(cell).iterator();}} (could this be done in a more efficient
way?)
while CollectionBackedScanner is simply continuing with the same existing
iterator
Which is the correct behaviour? could it be that we ask to reseek for a cell
that precedes the current cell during a flush? Why even call reseek(cell) and
not only next() during a flush?
In addition, the current patch only solves the performance issue for the
Segment class (in the no-IMC implementation). What about solving this also for
CompositeImmutableSegment? If we follow the code in the description and not in
the patch it should be an easy fix.
Regarding the performance results - do you see the same affect when running
with --writeToWAL=true?
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.001.patch, hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return
[jira] [Assigned] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eshcar Hillel reassigned HBASE-20628:
-
Assignee: (was: Eshcar Hillel)
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.001.patch, hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return Collections.singletonList(new
> CollectionBackedScanner(this.cellSet.get(), comparator));
> + }
> +
>/**
> * @return whether the segment has any cells
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eshcar Hillel reassigned HBASE-20628:
-
Assignee: Eshcar Hillel (was: stack)
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Assignee: Eshcar Hillel
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.001.patch, hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return Collections.singletonList(new
> CollectionBackedScanner(this.cellSet.get(), comparator));
> + }
> +
>/**
> * @return whether the segment has any cells
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel reassigned HBASE-20188: - Assignee: (was: Eshcar Hillel) > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel reassigned HBASE-20188: - Assignee: Eshcar Hillel (was: stack) > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: Eshcar Hillel >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Attachment: HBASE-20390.branch-2.0.003.patch > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Release Note: Changing in-memory compaction (IMC) defaults: (1) active segment is 1.4% of entire memstore (hbase.memstore.inmemoryflush.threshold.factor), (2) number of segments in compaction pipeline is 2 (hbase.hregion.compacting.pipeline.segments.limit) was: Changing in-memory compaction (IMC) defaults: (1) active segment is 2% of entire memstore (hbase.memstore.inmemoryflush.threshold.factor), (2) number of segments in compaction pipeline is 2 (hbase.hregion.compacting.pipeline.segments.limit) > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, HBase 2.0 > performance evaluation - throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467272#comment-16467272 ] Eshcar Hillel commented on HBASE-20188: --- Thanks Anoop (: Meanwhile I have opened HBASE-20542 to elaborate on the new solution. We can continue the discussion there. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Attachment: HBase 2.0 performance evaluation - throughput SSD_HDD.pdf > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, HBase 2.0 > performance evaluation - throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467243#comment-16467243 ] Eshcar Hillel commented on HBASE-20390: --- Following HBASE-20188 we realized in-memory compaction combined with MSLABs may suffer from heap under-utilization due to internal fragmentation. For example, setting the active segment threshold to A=0.02 means it stores 0.02*128MB=2.56MB. Each such 2.5MB segment utilizes 2 chunks (spanning *4MB*) which are carried in the compaction pipeline until the data is flushed to disk. Each 2.5MB data taking 4MB space means IMC heap utilization is roughly at 65%. Not ideal. We therefore experimented with A=0.014, namely active segment of size 1.8MB, which fits into a single chunk (leaving some space for overflow etc). Running workloadx+workloada+workloadc show improvement in performance in all these workloads wrt the default parameters of IMC (results are attached). We therefore suggest to set the IMC defaults to A=0.014, S=2. I will make a new patch. This number is obliviously working well with the current default chunk size. In parallel we are working on a new solution in HBASE-20542 to circumvent this problem regardless of the size of chunks, workload distribution or size of data written. > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, HBase 2.0 > performance evaluation - throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Issue Type: Sub-task (was: Task) Parent: HBASE-20188 > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Priority: Major > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20542) Better heap utilization for IMC with MSLABs
Eshcar Hillel created HBASE-20542: - Summary: Better heap utilization for IMC with MSLABs Key: HBASE-20542 URL: https://issues.apache.org/jira/browse/HBASE-20542 Project: HBase Issue Type: Task Reporter: Eshcar Hillel Following HBASE-20188 we realized in-memory compaction combined with MSLABs may suffer from heap under-utilization due to internal fragmentation. This jira presents a solution to circumvent this problem. The main idea is to have each update operation check if it will cause overflow in the active segment *before* it is writing the new value (instead of checking the size after the write is completed), and if it is then the active segment is atomically swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to the compaction pipeline. Later on the IMC deamon will run its compaction operation (flatten index/merge indices/data compaction) in the background. Some subtle concurrency issues should be handled with care. We next elaborate on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467229#comment-16467229 ] Eshcar Hillel commented on HBASE-20188: --- Hi, wanted to share some interesting insights and benchmark results. We tried to understand why the benefit of in-memory compaction decreases when MSLABs are used. We finally realized this is due to internal fragmentation that causes under utilization of the memory. For example, setting the active segment threshold to A=0.02 means it stores 0.02*128MB=2.56MB. Each such 2.5MB segment utilizes 2 chunks (spanning *4MB*) which are carried in the compaction pipeline until the data is flushed to disk. Each 2.5MB data taking 4MB space means IMC heap utilization is roughly at 65%. Not ideal. We therefore experimented with A=0.014, namely active segment of size 1.8MB, which fits into a single chunk (leaving some space for overflow etc). Running workloadx+workloada+workloadc show improvement in performance in all these workloads wrt the default parameters of IMC (results are attached [^HBase 2.0 performance evaluation - throughput SSD_HDD.pdf] ). While the new default improves performance we believe there are still cases where an overflow may cause using 2 chunks instead of one. We have and idea how to circumvent this problem. Suggesting to move in two phases: (1) in Jira HBASE-20390 set IMC default parameters to best utilize memory also when using MSLABs. (2) in a new Jira present and implement a solution that avoids the chunk overflow problem. In addition, we are also considering more optimization in HBASE-20480 that potentially reduces overhead of temporary cell objects in while searching in a CCM segment. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20188: -- Attachment: HBase 2.0 performance evaluation - throughput SSD_HDD.pdf > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20455) [IHC] Workloadx
[
https://issues.apache.org/jira/browse/HBASE-20455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445547#comment-16445547
]
Eshcar Hillel commented on HBASE-20455:
---
In your experiment with workloadx you use buffer size=2MB while the workload
was set to use buffer size=10KB
The values are also smaller so you need to run 100M ops and have 100M records
to write 25GB of data.
You can try using the shell script I posted HBASE-20188-xac.sh -- it pre-splits
the table which makes the experiment less "noisy".
If you want to have a bigger buffer size and run longer experiment you can just
run 500M ops.
> [IHC] Workloadx
> ---
>
> Key: HBASE-20455
> URL: https://issues.apache.org/jira/browse/HBASE-20455
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Fix For: 2.0.0
>
>
> I tried workloadx from parent issue, the ycsb workload that is meant to make
> IHC shine. I'm doing something wrong. I tried just plugging it in and
> doing this:
> {code}
> ycsb run hbase12 -P /home/stack/ycsb/workloads/workloadx -p table=ycsb
> -threads 48 -cp /home/stack/conf_hbase -p columnfamily=family -p
> clientbuffering=true -p writebuffersize=2097152 -s -p maxexecutiontime=1200
> -jvm-args=-Xmx8192
> m -Djava.security.egd=file:/dev/./urandom -p recordcount=2500 -p
> operationcount=2500 -p
> exportfile=logs/ycsb-workloadx-measurements-ve0524-20180419T02:58:14Z.json -p
> exporter=com.yahoo.ycsb.measurements.exporter.JSONArrayMeasurementsExporter
> {code}
> It completes in a minute and a half. Claims stuff like this for throughput:
> 236825 ops/second. In this case I have in-memory-compaction set to NONE. I
> was going to compare before and after.
> [~eshcar]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440582#comment-16440582 ] Eshcar Hillel commented on HBASE-20188: --- [~anoop.hbase] can you share the exact command line you are running. Can you report the throughput numbers for 1.4.2 and for 2.0.0. Thanks. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Attachment: HBASE-20390.branch-2.0.002.patch > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439140#comment-16439140 ] Eshcar Hillel commented on HBASE-20411: --- I briefly went over the code in RB. Did you had a chance to check if this patch boosts performance? > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 41901.lock.svg, > HBASE-20411.branch-2.0.001.patch, HBASE-20411.branch-2.0.002.patch, > HBASE-20411.branch-2.0.003.patch > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Release Note: Changing in-memory compaction (IMC) defaults: (1) active segment is 2% of entire memstore (hbase.memstore.inmemoryflush.threshold.factor), (2) number of segments in compaction pipeline is 2 (hbase.hregion.compacting.pipeline.segments.limit) Status: Patch Available (was: Open) > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Attachment: HBASE-20390-branch-2.0-01.patch > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437794#comment-16437794 ] Eshcar Hillel commented on HBASE-20188: --- [~anoop.hbase] can you post the command line or script + settings you are using so we can re-produce these results? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435236#comment-16435236 ] Eshcar Hillel commented on HBASE-20188: --- We also ran the same experiments in HDD machines and indeed we see that in HDD the lift of IMC over None is smaller. We now repeat all experiments again to eliminate noisy results. We will post a summary of both SSD and HDD results by EoD tomorrow. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434384#comment-16434384 ] Eshcar Hillel edited comment on HBASE-20188 at 4/12/18 9:25 AM: bq. I should add workloadx to the suite of tests I'm trying? Yes please do. In my script [^HBASE-20188-xac.sh] I run it after (pre)splitting into 10 regions. Opening a sub-jira to update defaults. was (Author: eshcar): vq. I should add workloadx to the suite of tests I'm trying? Yes please do. In my script [^HBASE-20188-xac.sh] I run it after (pre)splitting into 10 regions. Opening a sub-jira to update defaults. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20390: -- Issue Type: Sub-task (was: Task) Parent: HBASE-20188 > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Priority: Major > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel reassigned HBASE-20390: - Assignee: Eshcar Hillel > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20390) IMC Default Parameters for 2.0.0
Eshcar Hillel created HBASE-20390: - Summary: IMC Default Parameters for 2.0.0 Key: HBASE-20390 URL: https://issues.apache.org/jira/browse/HBASE-20390 Project: HBase Issue Type: Task Reporter: Eshcar Hillel Setting new default parameters for in-memory compaction based on performance tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434384#comment-16434384 ] Eshcar Hillel commented on HBASE-20188: --- vq. I should add workloadx to the suite of tests I'm trying? Yes please do. In my script [^HBASE-20188-xac.sh] I run it after (pre)splitting into 10 regions. Opening a sub-jira to update defaults. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433710#comment-16433710 ] Eshcar Hillel commented on HBASE-20188: --- And another thing, might be just a distraction but we think it is worth mentioning – Backward scans should run faster with IMC since the flat segments are simply arrays and it is very simple to go to prev in O(1), while in skip list prev is implemented by seeking the prev key which is done in O(log n). We don't have an easy way to verify this since YCSB does not support testing this kind of operation, but we are looking into it and will report if something interesting comes up. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433704#comment-16433704 ] Eshcar Hillel commented on HBASE-20188: --- re-attaching [^workloadx] from the most recent experiment > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433700#comment-16433700 ] Eshcar Hillel commented on HBASE-20188: --- Just to summarize the results again -- We see that in write workloads IMC improves performance; it delays flush to disk and hence reduces number of disk compaction. When values are small IMC reduces memory occupancy by reducing metadata size (regardless of workload distribution), when the distribution is skewed IMC reduces memory occupancy by eliminating data duplication (regardless of value size), when the values are big and workload is uniform IMC doesn't help. For reads IMC is either comparable or slightly worse than None (no in-memory compaction). In addition measures we did in past experiments show that IMC reduces write amplification, again due to reducing number of disk compaction. I am opening a new Jira to change the default to the parameters that showed best performance in the recent benchmarks. Namely, IMC policy = ADAPTIVE, active segment porition = 0.02, limit on number of segments in pipeline = 2. We are continuing with our experiments to see if any additional changes can help improve the performance. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20188: -- Attachment: workloadx > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx, > workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432783#comment-16432783 ] Eshcar Hillel commented on HBASE-20188: --- Attaching additional benchmark results where we update and read single column rows [^HBase 2.0 performance evaluation - 8GB(1).pdf] In workloadx (write-only) with a single (wider) column -- Adaptive outperforms None by 15%. In workloads a and c with a single wide column – Adaptive and None are comparable. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20188: -- Attachment: HBase 2.0 performance evaluation - 8GB(1).pdf > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431148#comment-16431148 ] Eshcar Hillel commented on HBASE-20188: --- I attached the results of some additional experiments we ran with 8GB heap. We created a new workload [^workloadx]. It is a write-only skewed distribution, with client side batching. Results show that IMC with 2% active and 2 pipeline segments has an advantage over none in write-only workloads. Script to run the experiments is here [^HBASE-20188-xac.sh], and relevant hbase-site configuration here [^hbase-site.xml] we would still like to investigate the read latency with IMC we have some direction we plan to explore; will come back with results. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB.pdf, HBase 2.0 > performance evaluation - Basic vs None_ system settings.pdf, > ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png, > ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png, > ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png, > ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20188: -- Attachment: hbase-site.xml workloadx HBASE-20188-xac.sh > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB.pdf, HBase 2.0 > performance evaluation - Basic vs None_ system settings.pdf, > ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png, > ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png, > ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png, > ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20188: -- Attachment: HBase 2.0 performance evaluation - 8GB.pdf > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426168#comment-16426168
]
Eshcar Hillel commented on HBASE-20188:
---
I am running hbase12 client
This is the code of the client
{code}
package com.yahoo.ycsb.db.hbase12;
/**
* HBase 1.2 client for YCSB framework.
*
* A modified version of HBaseClient (which targets HBase v1.2) utilizing the
* shaded client.
*
* It should run equivalent to following the hbase098 binding README.
*
*/
public class HBaseClient12 extends com.yahoo.ycsb.db.HBaseClient10 {
}
{code}
The difference from hbase10 is just in the pom.xml file I believe, which
includes shaded-client instead of hbase-client - could this be to blame?
{code}
com.yahoo.ycsb
hbase10-binding
${project.version}
org.apache.hbase
hbase-client
...
org.apache.hbase
hbase-shaded-client
${hbase12.version}
{code}
> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0
> performance evaluation - Basic vs None_ system settings.pdf,
> ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png,
> ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png,
> ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png,
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png,
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png,
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml,
> lock.127.workloadc.20180402T200918Z.svg,
> lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor
> that it is much slower, that the problem is the asyncwal writing. Does
> in-memory compaction slow us down or speed us up? What happens when you
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something
> about perf when 2.0.0 ships.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426134#comment-16426134
]
Eshcar Hillel commented on HBASE-20188:
---
This is the code that initiates {{bufferedMutator}} in ycsb:
{code:java}
final TableName tName = TableName.valueOf(table);
this.currentTable = connection.getTable(tName);
if (clientSideBuffering) {
final BufferedMutatorParams p = new BufferedMutatorParams(tName);
p.writeBufferSize(writeBufferSize);
this.bufferedMutator = connection.getBufferedMutator(p);
}
{code}
so need to understand why {{connection.getBufferedMutator(p)}} returns null
> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0
> performance evaluation - Basic vs None_ system settings.pdf,
> ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png,
> ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png,
> ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png,
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png,
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png,
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml,
> lock.127.workloadc.20180402T200918Z.svg,
> lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor
> that it is much slower, that the problem is the asyncwal writing. Does
> in-memory compaction slow us down or speed us up? What happens when you
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something
> about perf when 2.0.0 ships.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426130#comment-16426130
]
Eshcar Hillel commented on HBASE-20188:
---
I am trying to write a new workload that applies client side buffering by using
{{clientbuffering=true}}
However when running the workload I get the following exception in line
{{Preconditions.checkNotNull(bufferedMutator);}}
{code}
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
at com.yahoo.ycsb.db.HBaseClient10.update(HBaseClient10.java:441)
at com.yahoo.ycsb.DBWrapper.update(DBWrapper.java:198)
at
com.yahoo.ycsb.workloads.CoreWorkload.doTransactionUpdate(CoreWorkload.java:775)
at
com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:608)
at com.yahoo.ycsb.ClientThread.run(Client.java:454)
at java.lang.Thread.run(Thread.java:745)
{code}
I was able to use this ycsb property in the past.
Anyone aware of changes to client implementation that result in a null
{{bufferedMutator}}?
> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0
> performance evaluation - Basic vs None_ system settings.pdf,
> ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png,
> ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png,
> ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png,
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png,
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png,
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml,
> lock.127.workloadc.20180402T200918Z.svg,
> lock.2.memsize2.c.20180403T160257Z.svg, run_ycsb.sh, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor
> that it is much slower, that the problem is the asyncwal writing. Does
> in-memory compaction slow us down or speed us up? What happens when you
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something
> about perf when 2.0.0 ships.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425146#comment-16425146 ] Eshcar Hillel commented on HBASE-20188: --- bq. can you please share how can we switch on short-circuit reads in our experiments? OK I see how to do it in HBASE-20377. Do I need to set all 5 parameters that are mentioned there or just the first? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0 > performance evaluation - Basic vs None_ system settings.pdf, > ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png, > ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png, > ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png, > ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425142#comment-16425142
]
Eshcar Hillel commented on HBASE-20188:
---
{quote}Did you use hbase defaults or did you change segment count or flush size
from default?
{quote}
No I did not change any default except for the system settings (cms and mslab)
in the second and third experiment.
2 major differences with respect to your setting is (1) I run on SSD *and* (2)
I use only 8GB heap.
Are you still using 31GB heap in your runs? 31GB heap for 25GB of data is too
much. With 8GB I think the gc affect is more pronounced.
You can run an experiment with 0.02 (this was shown to be optimal once) but I
wouldn't haste in changing *any* default before we run full experiments. There
are several parameters that affect each other, as I mentioned above (pipeline
length, active portion CAM/CCM, etc.), and I would like to check all of them
more deeply, both in the current workloada/workloadc and in an additional
workload.
But before we run any further experiments, can you please share how can we
switch on short-circuit reads in our experiments? Thanks.
> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0
> performance evaluation - Basic vs None_ system settings.pdf,
> ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png,
> ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png,
> ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png,
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png,
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png,
> flamegraph-1072.1.svg, flamegraph-1072.2.svg,
> lock.127.workloadc.20180402T200918Z.svg,
> lock.2.memsize2.c.20180403T160257Z.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor
> that it is much slower, that the problem is the asyncwal writing. Does
> in-memory compaction slow us down or speed us up? What happens when you
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something
> about perf when 2.0.0 ships.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424604#comment-16424604 ] Eshcar Hillel commented on HBASE-20188: --- Attached are the results of evaluations over *SSD* machines [^HBase 2.0 performance evaluation - Basic vs None_ system settings.pdf] , and the script to run them [^HBASE-20188.sh] (which is based on the script by Stack). The setting is also similar: 1 master, 1RS with 8GB heap, 1 ycsb client, underlying HDFS set to 3-way replication. Comparing Basic with default configuration vs None under different system settings: cms/mslab vs cms/no-mslabs vs g1gc/no-maslab Summary of results: 1) None outperforms Basic in a uniform distribution of insert-only operations that includes multiple split events 2) Basic outperforms None in a mixed workload with zipfian distribution 3) None is slightly better than Basic in read-only zipfian workload 4) not using mslab improves performance in zipfian distribution workloads and has a negative effect with insert-only uniform workload 5) g1gc performs worse in all cases; this could be due to lack of tuning It is important to note that each configuration was tested once, each of these runs can be an outlier - a good or bad outlier Next we will come up with a workload which demonstrates the advantage of in-memory compaction as well as continue with benchmarks to determine optimal default values for in-memory compaction, namely portion of active segment, length of pipeline, etc. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0 > performance evaluation - Basic vs None_ system settings.pdf, > ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png, > ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png, > ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png, > ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20188: -- Attachment: HBASE-20188.sh > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188.sh, HBase 2.0 > performance evaluation - Basic vs None_ system settings.pdf, > ITBLL2.5B_1.2.7vs2.0.0_cpu.png, ITBLL2.5B_1.2.7vs2.0.0_gctime.png, > ITBLL2.5B_1.2.7vs2.0.0_iops.png, ITBLL2.5B_1.2.7vs2.0.0_load.png, > ITBLL2.5B_1.2.7vs2.0.0_memheap.png, ITBLL2.5B_1.2.7vs2.0.0_memstore.png, > ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20188: -- Attachment: HBase 2.0 performance evaluation - Basic vs None_ system settings.pdf > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, HBase 2.0 performance evaluation - > Basic vs None_ system settings.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging
[
https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422191#comment-16422191
]
Eshcar Hillel commented on HBASE-20259:
---
I also added a comment in RB – the description of Adaptive policy is missing.
I can add it if you like.
> Doc configs for in-memory-compaction and add detail to in-memory-compaction
> logging
> ---
>
> Key: HBASE-20259
> URL: https://issues.apache.org/jira/browse/HBASE-20259
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20259.master.001.patch,
> HBASE-20259.master.002.patch, HBASE-20259.master.003.patch
>
>
> I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems
> like in-memory is still on. My table looks like this:
> {code}
> Table ycsb is ENABLED
> ycsb
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false',
> NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE',
> CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER =
> > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false',
> > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false',
> > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
> {code}
> Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show
> in the above).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422176#comment-16422176 ] Eshcar Hillel commented on HBASE-20188: --- I agree the read performance degradation shown in https://docs.google.com/spreadsheets/d/1w2NBqAPFthG8Ib4C0pHpLARYpWoIF2Vck2vHZW77zE4/edit#gid=2102226724 is the number one problem to tackle. Any news with the directions you suggested Stack? bq. ASYNC_WAL does not work. SYNC_WAL is default. Can you elaborate on this? I still see the enum with ASYNC_WAL in the code. Do you mean there is another way to enable async wal? As far as I know this is a very common setting and we should enable it. bq. It looks like this will be the case, yes. We've not done the work to flip to G1GC. What would this (flip to g1gc) entail? HBASE-3455 is from 2011 surely some progress have been made since then. It would be unfortunate to get such a big release of HBase without adjusting to the progress in jvm management. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418735#comment-16418735
]
Eshcar Hillel commented on HBASE-20188:
---
One way to explain the poor performance of reads with in-memory compaction is
the fact that it uses 5 segments in the pipeline, and the fact that in the
performed benchmarks majority of keys did not have values on disk (only 15-20M
out of 100M keys are covered after the load phase) aggravates the negative
affect of these segments. But this is yet to be proved.
I will also try the 2.0 code with the same system settings and YCSB workloads,
except that I have SSD machines, but I believe that's ok.
Let me make sure I have all settings correct:
1) You run the code currently committed to *branch-2.0*
2) run on a single machine, namely *no replication* at the HDFS
3) master and RS are on the same machine
4) Heap size is *8GB*?
5) what is the {{operationcount}} in the experiments? (defined as
{{operationcount=$\{INSERT_COUNT}}})
I would like to run the tests until they are completed and not to cap them at
20minutes if that's ok.
I do understand that this will make the experiments much longer; can we agree
to use {{RECORD_COUNT=5000}} loading 50GB? This is less keys but more data
than in the experiments above.
What is the default for writing to WAL in branch-2.0, is it SYNC_WAL or
ASYNC_WAL?
> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
>Reporter: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: ITBLL2.5B_1.2.7vs2.0.0_cpu.png,
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png,
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png,
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png,
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png,
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png,
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor
> that it is much slower, that the problem is the asyncwal writing. Does
> in-memory compaction slow us down or speed us up? What happens when you
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something
> about perf when 2.0.0 ships.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20236) [locking] Write-time worst offenders
[
https://issues.apache.org/jira/browse/HBASE-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417228#comment-16417228
]
Eshcar Hillel commented on HBASE-20236:
---
I can take a look into this.
I see the semaphor used in {{FastPathBalancedQueueRpcExecutor}} is initiated
with zero permits, so acquire succeeds only after a release is executed,
otherwise the thread is blocked (parking). I am trying to understand why the
semaphor is needed what does it protect against. It is mostly related to
{{loadedCallRunner}} but there is no documentation in the code of what exactly
is going on. I wonder what happens if I simply remove the semaphor? [~stack]
you are the one signed-off this code, can you explain the code or knows someone
who can or have a pointer to a blog or any documentation?
> [locking] Write-time worst offenders
>
>
> Key: HBASE-20236
> URL: https://issues.apache.org/jira/browse/HBASE-20236
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Priority: Major
>
> Messing w/ my new toy, here are worst offenders locking; they must be bad if
> they show up in this sampling profiler:
> {code}
> 7 Total: 769321884622 (99.24%) samples: 2965
> 8 [ 0] java.util.concurrent.Semaphore$NonfairSync
> 9 [ 1] sun.misc.Unsafe.park
> 10 [ 2] java.util.concurrent.locks.LockSupport.park
> 11 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 12 [ 4]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly
> 13 [ 5]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
> 14 [ 6] java.util.concurrent.Semaphore.acquire
> 15 [ 7]
> org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner
> 16 [ 8] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 17
> 18 Total: 4284274263 (0.55%) samples: 23543
> 19 [ 0] org.apache.hadoop.hbase.regionserver.MutableSegment
> 20 [ 1] org.apache.hadoop.hbase.ByteBufferKeyValue.getSequenceId
> 21 [ 2] org.apache.hadoop.hbase.regionserver.Segment.updateMetaInfo
> 22 [ 3] org.apache.hadoop.hbase.regionserver.Segment.internalAdd
> 23 [ 4] org.apache.hadoop.hbase.regionserver.MutableSegment.add
> 24 [ 5] org.apache.hadoop.hbase.regionserver.AbstractMemStore.internalAdd
> 25 [ 6] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 26 [ 7] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 27 [ 8] org.apache.hadoop.hbase.regionserver.HStore.add
> 28 [ 9] org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore
> 29 [10] org.apache.hadoop.hbase.regionserver.HRegion.access$600
> 30 [11]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore
> 31 [12]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0
> 32 [13]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$442.1445825895.visit
> 33 [14]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations
> 34 [15]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore
> 35 [16]
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore
> 36 [17] org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate
> 37 [18] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 38 [19] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 39 [20] org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp
> 40 [21]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp
> 41 [22]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation
> 42 [23] org.apache.hadoop.hbase.regionserver.RSRpcServices.multi
> 43 [24]
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
> 44 [25] org.apache.hadoop.hbase.ipc.RpcServer.call
> 45 [26] org.apache.hadoop.hbase.ipc.CallRunner.run
> 46 [27] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 47 [28] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 48
> 49 Total: 717708856 (0.09%) samples: 214
> 50 [ 0] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
> 51 [ 1] sun.misc.Unsafe.park
> 52 [ 2] java.util.concurrent.locks.LockSupport.park
> 53 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 54 [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
> 55 [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
> 56 [ 6]
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412807#comment-16412807 ] Eshcar Hillel commented on HBASE-20188: --- Something to consider: When I ran the experiments to determine the optimal default values for in-memory compaction (HBASE-16417), I ran them with CAM setting and no mslab. My findings for the optimal setting (for SSD) were a threshold of 2% for the active segment (hbase.memstore.inmemoryflush.threshold.factor=0.02) and pipeline length of 4 (hbase.hregion.compacting.pipeline.segments.limit=4), and 3 for hdd. *However*, the current settings are different: 1) using mslab (which means CCM setting and not CAM) 2) the active segment threshold was changed and is now hbase.memstore.inmemoryflush.threshold.factor=0.1 But the pipeline length is still set to 4. This doesn't make much sense. If we flush "only" after aggregating 10% of the data in the active segment we can use a pipeline of length 1, this would be equivalent to a longer pipeline (of length 4) with smaller segments (consisting of 2% of the data). I hope what I am saying here makes sense. Bottom line, given that the threshold was changed to 10%, I would recommend changing the pipeline length to 1. This should mainly have a positive affect on read latency. And regardless, presuming mslab is default (which is not what we recommend) may require an additional round of parameter tuning. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412792#comment-16412792 ] Eshcar Hillel commented on HBASE-20188: --- Stack the table with throughput and latency numbers you posted is very helpful (looking at the figures, it is much harder to say which settings are at each run). Can you give the same table with the none numbers? BTW -- do you have the 50th and 99th latency percentiles (mainly for reads)? They are much more informative than the average latency which is somewhere between them. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409243#comment-16409243 ] Eshcar Hillel commented on HBASE-20188: --- What does load (in YCSB_load) stands for? Which distribution are you using in the YCSB runs uniform/zipfian/something else? What is the size of values? How many regions? columns? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_MEMSTORE.png, YCSB_OPs.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406124#comment-16406124
]
Eshcar Hillel commented on HBASE-19639:
---
I might have misinterpreted a line in the log. How would you interpret the
following lines:
{code:java}
regionserver.ChunkCreator: data allocating 2 MB MemStoreChunkPool with chunk
size 2830, max count 0, initial count {}
regionserver.ChunkCreator: index allocating 204.80 KB MemStoreChunkPool with
chunk size 3145, max count 0, initial count {}
{code}
Maybe it means there is a pool but no chunk is pre-allocated?
Anyway, I would strongly suggest to compare performance to a setting where G1GC
is used and no MSLAB.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
>
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406022#comment-16406022
]
Eshcar Hillel commented on HBASE-19639:
---
Thanks [~stack]
I went over the log - didn't see any {{RegionTooBusyException}} and most
flushes are when the memstore size is ~128MB, these are good news I guess...
However, here are 2 comments w.r.t settings:
1. you use mark GC - might consider G1GC
2. you are using MSLAB with no pool, better to use chunk pool with MSLAB, and
best not to work with MSLAB at all ;)
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO
