[
https://issues.apache.org/jira/browse/PHOENIX-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127229#comment-18127229
]
Tanuj Khurana commented on PHOENIX-8023:
----------------------------------------
Another stack trace this time when adding index to the parent's PTable
{code:java}
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:256)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:54)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:280)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:450)
at
org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:324)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:623)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTTLForParentTable(MetaDataEndpointImpl.java:1915)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1785)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1154)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1149)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:809)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:4085)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.addIndexToTable(MetaDataEndpointImpl.java:910)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1652)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1154)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1149)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:809)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:4085)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:728)
at
org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:19507)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8011)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2540)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2514)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45035)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:416)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
> SYSTEM.CATALOG region unavailable for entire scan duration when closing
> -----------------------------------------------------------------------
>
> Key: PHOENIX-8023
> URL: https://issues.apache.org/jira/browse/PHOENIX-8023
> Project: Phoenix
> Issue Type: Bug
> Reporter: Tanuj Khurana
> Priority: Major
>
> _MetaDataEndpointImpl.buildTable_ holds the SYSTEM.CATALOG region operation
> (read lock)
> while building a PTable, and on that path the TTL resolution helpers
> (getTTLFromHierarchy / getTTLForParentTable) introduced as part of
> PHOENIX-6978 open a HBase client scan of SYSTEM.CATALOG itself via
> {_}ServerUtil.getHTableForCoprocessorScan(env, SYSTEM.CATALOG){_}. Once a
> close sets the
> closing flag, those nested scans get '{*}NotServingRegionException ... is
> closing{*}' and the HBase client *retries for the full 60 s scanner timeout*
> while the outer handler still holds the region read lock. The close cannot
> take the write lock until they give up and the scan operation times out.
>
> {code:java}
> java.net.SocketTimeoutException: callTimeout=60000, callDuration=68468:
> SYSTEM.CATALOG,,1787178855993.695f91ca... is closing ...
> hostname=regionserver-sys-2...
> at
> o.a.h.h.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(...:294)
> at o.a.h.h.client.ScannerCallableWithReplicas.call(...:256)
> at o.a.h.h.client.ClientScanner.next(ClientScanner.java:623)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTTLFromHierarchy(MetaDataEndpointImpl.java:1837)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1571)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1154)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1149)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:809)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:4085)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:728)
> at ...MetaDataProtos$MetaDataService.callMethod(...)
> at o.a.h.h.regionserver.HRegion.execService(HRegion.java:8011)
> ...
> at o.a.h.h.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: java.net.SocketTimeoutException: callTimeout=60000,
> callDuration=68468: ... is closing
> at
> o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157)
> at o.a.h.h.client.ResultBoundedCompletionService$QueueingFuture.run(...:74)
> Caused by: org.apache.hadoop.hbase.NotServingRegionException:
> SYSTEM.CATALOG,... is closing
> at o.a.h.h.regionserver.HRegion.startRegionOperation(HRegion.java:8130)
> at o.a.h.h.regionserver.HRegion.getScanner(HRegion.java:3059)
> at
> o.a.h.h.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3205)
> at o.a.h.h.regionserver.RSRpcServices.scan(RSRpcServices.java:3578)
> at
> o.a.h.h.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.scan(...:149)
> at o.a.h.h.client.ScannerCallable.openScanner(ScannerCallable.java:363)
> ...
> at
> o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:103){code}
>
> This happens twice because on a graceful shutdown syscat region first moves
> from the original regionserver to another regionserver and then again moves
> back to the original regionserver.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
