Viraj Jasani created PHOENIX-7363:
-------------------------------------
Summary: All meta handlers on SYSTEM.CATALOG regionserver
exhausted within CQSI#init
Key: PHOENIX-7363
URL: https://issues.apache.org/jira/browse/PHOENIX-7363
Project: Phoenix
Issue Type: Bug
Reporter: Viraj Jasani
After allowing readLock for getTable API at the server side (PHOENIX-6066), it
seems that under heavy load, all meta handler threads can get exhausted within
ConnectionQueryServicesImpl initialization as part of any of the
MetaDataEndpointImpl coproc operations. When the table details are not present
in the cache, MetaDataEndpointImpl coproc can attempt to create new connection
on the server side in order to scan SYSTEM.CATALOG. Under heavy load, several
(all of) meta handlers – which are dedicated for all metadata (system table)
operations – could attempt to create server side connection, which can further
lead into creating new PhoenixConnection to execute CREATE TABLE DDL for
SYSTEM.CATALOG in order to ensure that the System tables exist.
{code:java}
"RpcServer.Metadata.Fifo.handler=254,queue=20,port=60020" #927 daemon prio=5
os_prio=0 tid=0x00007fd53f16a000 nid=0x473 waiting for monitor entry
[0x00007fd4b1234000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3547)
- waiting to lock <0x000000047da00058> (a
org.apache.phoenix.query.ConnectionQueryServicesImpl)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3537)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3537)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:272)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:229)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:433)
at
org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:410)
at
org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:391)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1499)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.addIndexToTable(MetaDataEndpointImpl.java:832)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1490)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:669)
at
org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:19507)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7941)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2537)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2511)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45035)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
{code:java}
"RpcServer.Metadata.Fifo.handler=142,queue=12,port=60020" #815 daemon prio=5
os_prio=0 tid=0x00007fd53f07f000 nid=0x403 waiting on condition
[0x00007fd4b8234000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000047d400640> (a
java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.hadoop.hbase.client.HTable.lambda$coprocessorService$23(HTable.java:1150)
at org.apache.hadoop.hbase.client.HTable$$Lambda$933/100908000.run(Unknown
Source)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1128)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1114)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:1855)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1601)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:2201)
at
org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:3202)
at
org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1028)
at
org.apache.phoenix.compile.CreateTableCompiler$CreateTableMutationPlan.execute(CreateTableCompiler.java:483)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:569)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:534)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:533)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:521)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:2235)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3586)
- locked <0x000000047da00058> (a
org.apache.phoenix.query.ConnectionQueryServicesImpl)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3537)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3537)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:272)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:229)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:433)
at
org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:410)
at
org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:391)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1499)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.addIndexToTable(MetaDataEndpointImpl.java:832)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1490)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
at
org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:669)
at
org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:19507)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7941)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2537)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2511)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45035)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
The system tables existence check is not required for server side connections.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
