[ https://issues.apache.org/jira/browse/PHOENIX-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310280#comment-16310280 ]
Francis Chuang commented on PHOENIX-4374: ----------------------------------------- I am not very familiar with Java and log4j. Is the full config key `log4j.logger.org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure=trace`? > Flakyness with Phoenix 4.13.0 and HBase 1.3.1: RuntimeException: > org.apache.phoenix.exception.PhoenixIOException: > org.apache.hadoop.hbase.NamespaceNotFoundException: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: PHOENIX-4374 > URL: https://issues.apache.org/jira/browse/PHOENIX-4374 > Project: Phoenix > Issue Type: Bug > Affects Versions: 4.13.0 > Reporter: Francis Chuang > Attachments: 2017-12-29.m9edd51.log, 2018-01-03.m9edd51.log, > hbase--master-m9edd51-phoenix.m9edd51-2017-11-20-second-run.log, > hbase--master-m9edd51-phoenix.m9edd51-2017-11-20.log, > hbase--master-m9edd51-phoenix.m9edd51.log, > protobufs-dump-2017-11-20-second-run.zip, protobufs-dump-2017-11-20.zip, > root-queryserver-2017-11-20-second-run.log, root-queryserver-2017-11-20.log, > root-queryserver.log, > tephra-service--m9edd51-phoenix.m9edd51-2017-11-20-second-run.log, > tephra-service--m9edd51-phoenix.m9edd51-2017-11-20.log, > tephra-service--m9edd51-phoenix.m9edd51.log > > > I am using the Phoenix Query Server via my [Go Avatica SQL > driver|https://github.com/Boostport/avatica]. > In terms of my set up I am running Phoenix 4.13.0 and HBase 1.3.1 in docker > with a single node HBase using local storage. The dockerfile is available > here: https://github.com/Boostport/hbase-phoenix-all-in-one > Today, I updated one of my projects to use the latest version of the above > image (Phoenix 4.13.0 and HBase 1.3.1) and my integration tests against > Phoenix + HBase have become extremely flaky. The tests use a mix of > transactional and non-transactional tables. > The flakyness is that random tests will fail with the same error. If I rerun > the tests, they sometimes pass and sometimes fail, and it is not clear why > this is happening. > In all of these tests, I am doing the following: > 1. Create the schema. > 2. Create tables. > 3. Insert, delete and read data. > 4. Delete the tables and schema. > This is the error I get when trying to drop the schema: > {code:java} > An error was encountered while processing your request: RuntimeException: > org.apache.phoenix.exception.PhoenixIOException: > org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at > org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45) > at > org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114) > at > org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85) > at > org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717) > at > org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133) > at > org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705) > at > org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496) > at > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601) > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339) > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException): > INITTEST > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49) > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139) > at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498) > -> PhoenixIOException: > org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at > org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45) > at > org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114) > at > org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85) > at > org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717) > at > org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133) > at > org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705) > at > org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496) > at > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601) > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339) > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException): > INITTEST > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49) > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139) > at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498) > -> NamespaceNotFoundException: > org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at > org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45) > at > org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114) > at > org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85) > at > org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717) > at > org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133) > at > org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705) > at > org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496) > at > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601) > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339) > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException): > INITTEST > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49) > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139) > at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498) > -> RemoteWithExtrasException: > org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at > org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45) > at > org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114) > at > org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85) > at > org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717) > at > org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133) > at > org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705) > at > org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496) > at > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601) > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339) > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException): > INITTEST > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83) > at > org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49) > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139) > at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498) > {code} > I have also attached the following: > - Logs from tephra > - Logs from PQS/Avatica > - Logs from HBase > Further update: > Rolled back to the HBase 1.2 Phoenix 4.11 image and tests are rock solid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)