[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135462#comment-15135462 ]
stack commented on HBASE-15213: ------------------------------- Branch-1.1 has javadoc issues to be cleaned up and it has more flakey tests. > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > ----------------------------------------------------------------------- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Sub-task > Components: Performance > Reporter: Junegunn Choi > Assignee: Junegunn Choi > Fix For: 1.1.4, 1.0.4 > > Attachments: 15157v3.branch-1.1.patch, HBASE-15213-increment.png, > HBASE-15213.branch-1.0.patch, HBASE-15213.v1.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1 | 2661 | 2486 | 2359 | > || 2 | 5048 | 5064 | 4867 | > || 4 | 7503 | 8071 | 8690 | > || 8 | 10471 | 10886 | 13980 | > || 16 | 15515 | 9418 | 18601 | > || 32 | 17699 | 5421 | 20540 | > || 64 | 20601 | 4038 | 25591 | > || 96 | 19177 | 3891 | 26017 | > We can clearly observe that the throughtput degrades as we increase the > number of concurrent requests, which led me to believe that there's severe > context switching overhead and I could indirectly confirm that suspicion with > cs entry in vmstat output. branch-1.0 shows a much higher number of context > switches even with much lower throughput. > Here are the observations: > - WriteEntry in the writeQueue can only be removed by the very handler that > put it, only when it is at the front of the queue and marked complete. > - Since a WriteEntry is marked complete after the wait-loop, only one entry > can be removed at a time. > - This stringent condition causes O(N^2) context switches where n is the > number of concurrent handlers processing requests. > So what I tried here is to mark WriteEntry complete before we go into > wait-loop. With the change, multiple WriteEntries can be shifted at a time > without context switches. I changed writeQueue to LinkedHashSet since fast > containment check is needed as WriteEntry can be removed by any handler. > The numbers look good, it's virtually identical to pre-HBASE-8763 era. > || Num threads || branch-1.0 with fix || > || 1 | 2459 | > || 2 | 4976 | > || 4 | 8033 | > || 8 | 12292 | > || 16 | 15234 | > || 32 | 16601 | > || 64 | 19994 | > || 96 | 20052 | > So what do you think about it? Please let me know if I'm missing anything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)