[
https://issues.apache.org/jira/browse/HUDI-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Sumit updated HUDI-3983:
------------------------------
Status: In Progress (was: Open)
> ClassNotFoundException when using hudi-spark-bundle to write table with hbase
> index
> -----------------------------------------------------------------------------------
>
> Key: HUDI-3983
> URL: https://issues.apache.org/jira/browse/HUDI-3983
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: xi chaomin
> Assignee: Sagar Sumit
> Priority: Critical
> Fix For: 0.12.1
>
>
> I ran a spark job and encountered several ClassNotFoundExceptions. spark
> version is 3.1 and scala version is 2.12.
> 1.
> {code:java}
> java.lang.NoClassDefFoundError:
> org/apache/hudi/org/apache/hadoop/hbase/protobuf/generated/AuthenticationProtos$TokenIdentifier$Kind
> at
> org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:222)
> at
> org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:195)
> at
> org.apache.hudi.org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:395)
> at
> org.apache.hudi.org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
> at
> org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108)
> {code}
> including org.apache.hbase:hbase-protocol in
> packaging/hudi-spark-bundle/pom.xml can solve this error.
> 2.
> {code:java}
> java.lang.ClassNotFoundException:
> org.apache.hudi.org.apache.hbase.thirdparty.com.google.gson.GsonBuilder
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}
> including org.apache.hbase.thirdparty:hbase-shaded-gson n
> packaging/hudi-spark-bundle/pom.xml can solve this error.
> 3.
> {code:java}
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not
> found {code}
> There is a configuration in hbase-site.xml
> {code:java}
> <property>
> <name>hbase.status.listener.class&amp;amp;amp;amp;lt;/name>
>
> <value>org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener</value>
> <description>
> Implementation of the status listener with a multicast message.
> </description>
> </property> {code}
> I set _*hbase.status.listener.class*_ to
> _*org.apache.hudi.org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener*_
> in hbase configureation, the ClassNotFoundException has resolved, but get
> another exception
> {code:java}
> org.apache.hudi.org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=1, exceptions:
> 2022-08-26T07:12:57.603Z,
> RpcRetryingCaller{globalStartTime=2022-08-26T07:12:56.651Z, pause=100,
> maxAttempts=1},
> org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Call to address=x.x.x.x/x.x.x.x:16020 failed on local exception:
> org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Connection closed at
> org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:146)
> at
> org.apache.hudi.org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Call to address=x.x.x.x/x.x.x.x:16020 failed on local exception:
> org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Connection closed
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:214)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:384)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:89)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:415)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:411)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:118)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.Call.setException(Call.java:133)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:203)
> at
> org.apache.hudi.org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:211)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> Caused by:
> org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException:
> Connection closed
> ... 26 more{code}
> I remove the relocations related to hbase in
> packaging/hudi-spark-bundle/pom.xml , the job succeed.
> I checked the debug logs, but have no idea of the reason why
> ConnectionClosedException occurs when we use relocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
