[ https://issues.apache.org/jira/browse/HBASE-24515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127835#comment-17127835 ]
Toshihiro Suzuki commented on HBASE-24515: ------------------------------------------ Pushed to master, branch-2, branch-2.3, branch-2.2 and branch-2.1. [~vjasani] Thank you for the review! > batch Increment/Append fails when retrying the RPC > -------------------------------------------------- > > Key: HBASE-24515 > URL: https://issues.apache.org/jira/browse/HBASE-24515 > Project: HBase > Issue Type: Bug > Reporter: Toshihiro Suzuki > Assignee: Toshihiro Suzuki > Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.1.10, 2.2.6 > > > When a client hits RPC timeout and sends a second RPC request for batch > Increment/Append but the first RPC is already processed actually, the nonce > of the RPC is saved in the RS. > In this case, for the second RPC, the RS just reads the previous result and > returns it to the client to avoid the Increment/Append is processed twice. > At that time, for batch Increment/Append, we try to create a Get object from > a CellScanner object in the following code: > > [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776] > However, the CellScanner object is already consumed to create the > Increment/Append object in the following code, and it fails with the > following exception: > > [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757] > {code:java} > 2020-06-06 14:09:06,153 WARN [hconnection-0x79c3903e-shared-pool3-t209] > client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36, > failureCount=1ops, last > exception=org.apache.hadoop.hbase.DoNotRetryIOException: > org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0 > no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp: > 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to: > 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765 > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)