Viraj Jasani created PHOENIX-6758:
-------------------------------------
Summary: During HBase 2 upgrade Phoenix Self healing task fails to
create server side connection before reading SYSTEM.TASK
Key: PHOENIX-6758
URL: https://issues.apache.org/jira/browse/PHOENIX-6758
Project: Phoenix
Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani
Fix For: 5.2.0, 5.1.3
When HBase master is running on 1.x version and regionservers are on 2.x
version (specifically system rsgroup regionservers), TaskRegionObserver fails
to initiate connection to read SYSTEM.TASK records. While this task itself is
not a customer facing usecase, but we should fix the task being unable to
initiate connection for the first time. Once the connection is created, it is
anyways cached in CQSI.
Detailed stacktrace:
{code:java}
ERROR [pool-54-thread-2] coprocessor.TaskRegionObserver: SelfHealingTask failed!
org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family
table does not exist in region hbase:meta,,1.1588230740 in table 'hbase:meta',
{TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 =>
'|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}},
{NAME => 'info', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE',
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', IN_MEMORY => 'true',
COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192', METADATA =>
{'CACHE_DATA_IN_L1' => 'true'}}
at
org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:8476)
at
org.apache.hadoop.hbase.regionserver.HRegion.prepareGet(HRegion.java:8008)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2586)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2530)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45815)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
at
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:138)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1542)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1936)
at
org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:3084)
at
org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1111)
at
org.apache.phoenix.compile.CreateTableCompiler$CreateTableMutationPlan.execute(CreateTableCompiler.java:420)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:415)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:397)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:396)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:384)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1906)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3290)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3253)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3253)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:144)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:422)
at
org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:400)
at
org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:381)
at
org.apache.phoenix.coprocessor.TaskRegionObserver$SelfHealingTask.run(TaskRegionObserver.java:162)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
We should support fallback for admin#tableExists for Phoenix server side
dealing of CQSI#init.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
