We were running klyin 1.5 on CDH 5.7 and was working fine.
Post upgrade to CDH 5.9, we are seeing an exception while running the queries
in Kylin.
I have restarted the Kylin 1.5 instance and also build the test cube from
scratch. The cube built was successful but seeing an exception in the Kylin.log
when running the queries (in the Kyiin ‘Insight’ tab).
Please help us to resolve this issue at the earliest. Let us know if you need
any other information.
Kylin Version: 1.5.2
Excerpts from the Server Environment
HBASE_LOG_DIR = /var/log/hbase
HBASE_HOME = /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.21/lib/hbase/bin/..
HBASE_OPTS = -Xmx268435456 -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
-Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase.log
-Dhbase.home.dir=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.21/lib/hbase/bin/..
-Dhbase.id.str=hbase -Dhbase.root.logger=INFO,console
-Djava.library.path=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.21/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.21/lib/hbase/bin/../lib/native/Linux-amd64-64
-Dhbase.security.logger=INFO,NullAppender
HBASE_ENV_INIT = true
hbase_dependency =
/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.21/lib/hbase/bin/../lib/hbase-common-1.2.0-cdh5.9.0.jar
Kylin log exceprts
2017-01-26 18:33:50,419 INFO [http-bio-10101-exec-7] v2.CubeHBaseRPC:271 :
Visiting hbase table KYLIN_XXXXXXXXX: cuboid exact match, from 2 to 2 Start:
\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x02\x00 (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00) Stop:
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\xFF\x00
(\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\xFF\x00),
No Fuzzy Key
2017-01-26 18:33:50,419 DEBUG [http-bio-10101-exec-7]
v2.CubeHBaseEndpointRPC:319 : Submitting rpc to 1 shards starting from shard 0,
scan range count 1
2017-01-26 18:33:50,419 INFO [http-bio-10101-exec-7]
v2.CubeHBaseEndpointRPC:102 : Timeout for ExpectedSizeIterator is: 198000
2017-01-26 18:33:50,420 DEBUG [http-bio-10101-exec-7]
enumerator.OLAPEnumerator:127 : return TupleIterator...
2017-01-26 18:33:50,430 ERROR [pool-7-thread-5]
util.LoggableCachedThreadPool:44 : Execution exception when running task in
pool-7-thread-5
2017-01-26 18:33:50,431 ERROR [pool-7-thread-5]
util.LoggableCachedThreadPool:54 : Caught exception in thread pool-7-thread-5:
java.lang.RuntimeException: <sub-thread for GTScanRequest 1c42a42b> Error when
visiting cubes by endpoint
at
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:344)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment.getRegion()Lorg/apache/hadoop/hbase/regionserver/HRegion;
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment.getRegion()Lorg/apache/hadoop/hbase/regionserver/HRegion;
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:177)
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:4164)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7797)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1982)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1964)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
... 3 more
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1631)
at
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
at
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
at
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
at
org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:4225)
at
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:392)
at
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:388)
at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1797)
... 4 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment.getRegion()Lorg/apache/hadoop/hbase/regionserver/HRegion;
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment.getRegion()Lorg/apache/hadoop/hbase/regionserver/HRegion;
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:177)
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:4164)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7797)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1982)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1964)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
... 3 more
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
... 13 more
Thanks,
Manoj Jain
