[ https://issues.apache.org/jira/browse/HBASE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cosmin Lehene updated HBASE-3382: --------------------------------- Issue Type: Improvement (was: Bug) > Make HBase client work better under concurrent clients > ------------------------------------------------------ > > Key: HBASE-3382 > URL: https://issues.apache.org/jira/browse/HBASE-3382 > Project: HBase > Issue Type: Improvement > Components: Performance > Reporter: ryan rawson > Assignee: ryan rawson > Labels: delete > Attachments: HBASE-3382-nio.txt, HBASE-3382.txt > > > The HBase client uses 1 socket per regionserver for communication. This is > good for socket control but potentially bad for latency. How bad? I did a > simple YCSB test that had this config: > readproportion=0 > updateproportion=0 > scanproportion=1 > insertproportion=0 > fieldlength=10 > fieldcount=100 > requestdistribution=zipfian > scanlength=300 > scanlengthdistribution=zipfian > I ran this with 1 and 10 threads. The summary is as so: > 1 thread: > [SCAN] Operations 1000 > [SCAN] AverageLatency(ms) 35.871 > 10 threads: > [SCAN] Operations 1000 > [SCAN] AverageLatency(ms) 228.576 > We are taking a 6.5x latency hit in our client. But why? > First step was to move the deserialization out of the Connection thread, this > seemed like it could have a big win, an analog change on the server side got > a 20% performance improvement (already commited as HBASE-2941). I did this > and got about a 20% improvement again, with that 228ms number going to about > 190 ms. > So I then wrote a high performance nanosecond resolution tracing utility. > Clients can flag an API call, and we get tracing and numbers through the > client pipeline. What I found is that a lot of time is being spent in > receiving the response from the network. The code block is like so: > NanoProfiler.split(id, "receiveResponse"); > if (LOG.isDebugEnabled()) > LOG.debug(getName() + " got value #" + id); > Call call = calls.get(id); > size -= 4; // 4 byte off for id because we already read it. > ByteBuffer buf = ByteBuffer.allocate(size); > IOUtils.readFully(in, buf.array(), buf.arrayOffset(), size); > buf.limit(size); > buf.rewind(); > NanoProfiler.split(id, "setResponse", "Data size: " + size); > I came up with some numbers: > 11726 (receiveResponse) split: 64991689 overall: 133562895 Data size: 4288937 > 12163 (receiveResponse) split: 32743954 overall: 103787420 Data size: 1606273 > 12561 (receiveResponse) split: 3517940 overall: 83346740 Data size: 4 > 12136 (receiveResponse) split: 64448701 overall: 203872573 Data size: 3570569 > The first number is the internal counter for keeping requests unique from > HTable on down. The numbers are in ns, the data size is in bytes. > Doing some simple calculations, we see for the first line we were reading at > about 31 MB/sec. The second one is even worse. Other calls are like: > 26 (receiveResponse) split: 7985400 overall: 21546226 Data size: 850429 > which is 107 MB/sec which is pretty close to the maximum of gige. In my set > up, the ycsb client ran on the master node and HAD to use network to talk to > regionservers. > Even at full line rate, we could still see unacceptable hold ups of unrelated > calls that just happen to need to talk to the same regionserver. > This issue is about these findings, what to do, how to improve. -- This message was sent by Atlassian JIRA (v6.3.4#6332)