[
https://issues.apache.org/jira/browse/HBASE-29390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-29390.
-------------------------------
Fix Version/s: 2.7.0
3.0.0-beta-2
2.6.3
2.5.12
Hadoop Flags: Reviewed
Resolution: Fixed
Pushed to all active branches.
Thanks [~nihaljain.cs] for reviewing!
> Too many logs in AsyncBatchRpcRetryingCaller when hitting
> RegionTooBusyException
> --------------------------------------------------------------------------------
>
> Key: HBASE-29390
> URL: https://issues.apache.org/jira/browse/HBASE-29390
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.3, 2.5.12
>
>
> When checking the iTBLL result, I saw bunch of logs in the link generator
> logs like this
> {noformat}
> 2025-06-10 06:52:30,800 WARN [RPCClient-NioEventLoopGroup-1-8]
> org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller: Process batch for
> ['IntegrationTestBigLinkedList,<"\x0F\x91\xF3\x07\x83\x07\xDBI\xF8\xB5\xCA\xD6\x19\xB8,1749399341229.6a9162
> 4d74729cdd03e61ee7500e6951.'] in IntegrationTestBigLinkedList from
> data01,16020,1749509062898 failed, tries=6
> org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 M,
> regionName=6a91624d74729cdd03e61ee7500e6951, server=data01,16020,1749509062898
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:5379)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4840)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4766)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:939)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:904)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2877)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45448)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:457)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> at
> jdk.internal.reflect.GeneratedConstructorAccessor13.newInstance(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1725)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ResponseConverter.getResults(ResponseConverter.java:172)
> at
> org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller.lambda$sendToServer$11(AsyncBatchRpcRetryingCaller.java:409)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:56)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:47)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:400)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:430)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
> at org.apache.hadoop.hbase.ipc.Call.setResponse(Call.java:149)
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.finishCall(RpcConnection.java:396)
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.readResponse(RpcConnection.java:461)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:125)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:140)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> {noformat}
> The same output repeated many many times, which makes it really difficult to
> find useful logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
