In HRegion#increment(), we lock the row (not region):
try {
rowLock = getRowLock(row);
Can you pastebin the complete stack trace ?
Thanks
On Tue, Sep 8, 2015 at 2:01 AM, 鈴木俊裕 <brfrn...@gmail.com> wrote:
> Hi,
>
> We upgraded our cluster from CDH5.3.1(HBase0.98.6) to CDH5.4.5(HBase1.0.0)
> and we experience slowdown in increment operation.
>
> Here's an extract from thread dump of the RegionServer of our cluster:
>
> Thread 68 (RW.default.writeRpcServer.handler=15,queue=5,port=60020):
> State: BLOCKED
> Blocked count: 21689888
> Waited count: 39828360
> Blocked on java.util.LinkedList@3474e4b2
> Blocked by 63 (RW.default.writeRpcServer.handler=10,queue=0,port=60020)
> Stack:
> java.lang.Object.wait(Native Method)
>
>
> org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:224)
>
>
> org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:203)
>
> org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:6712)
>
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:501)
>
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:570)
>
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1901)
>
>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31451)
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
>
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> java.lang.Thread.run(Thread.java:745)
>
> There are many similar threads in the thread dump.
>
> I read the source code and I think this is caused by changes of
> MultiVersionConsistencyControl.
> A region lock (not a row lock) seems to occur in
> waitForPreviousTransactionsComplete().
>
>
> Also we wrote performance test code for increment operation that included
> 100 threads and ran it in local mode.
>
> The result is shown below:
>
> CDH5.3.1(HBase0.98.6)
> Throughput(op/s): 12757, Latency(ms): 7.975072509210629
>
> CDH5.4.5(HBase1.0.0)
> Throughput(op/s): 2027, Latency(ms): 49.11840157868772
>
>
> Thanks,
>
> Toshihiro Suzuki
>
