Ankit Singhal created PHOENIX-6331: -------------------------------------- Summary: Increase index retry from 1 to 2 incase of NotServingRegionException Key: PHOENIX-6331 URL: https://issues.apache.org/jira/browse/PHOENIX-6331 Project: Phoenix Issue Type: Bug Reporter: Ankit Singhal
Currently, we move the index to PENDING_DISABLE whenever the single write to index failed and carry out a retry at the client but this can be optimized for NotServingRegionException as Index regions can move very frequently depending on the balancer and one more retry at the server could avoid unnecessary handling of index states and retries at the client. {code:java} 2021-01-20 06:54:58,682 WARN org.apache.hadoop.hbase.client.AsyncProcess: #277, table=<index_table>, attempt=1/1 failed=1ops, last exception: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region <region> is not online on <region_server> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2997) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1069) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2100) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163) on <region_server>, tracking started Wed Jan 20 06:54:58 CET 2021; not retrying 1 - final failure 2021-01-20 06:54:58,690 INFO org.apache.phoenix.index.PhoenixIndexFailurePolicy: Successfully update INDEX_DISABLE_TIMESTAMP for <index_table> due to an exception while writing updates. indexState=PENDING_DISABLE org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: disableIndexOnFailure=true, Failed to write to multiple index tables: [<index_table>] at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:236) at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195) at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156) at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145) at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:617) at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:577) at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:560) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3421) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2944) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2886) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:765) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:716) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2146) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163) 2021-01-20 06:54:58,691 INFO org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=true, Failed to write to multiple index tables: [<index_table>] ,serverTimestamp=1611122098649, 2021-01-20 06:55:01,296 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for hbase (auth:SIMPLE) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)