?????? Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
Thanks for your reply. Can you describe it in more detail? Which dependency mismatch? It works well sometimes, but somtimes fails becaus of error 'NoSuchMethodError'. Thanks. -- -- ??: "vaquar khan";; : 2017??7??17??(??) 9:38 ??: ""<441586...@qq.com>; ????: "User"; : Re: Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel; Following error we are getting because of dependency mismatch. Regards, vaquar khan On Jul 17, 2017 3:50 AM, "zzcclp" <441586...@qq.com> wrote: Hi guys: I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with following command, error 'NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;' appears sometimes: command: *su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master yarn --deploy-mode client --files /opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g --num-executors 2 --executor-memory 8g --executor-cores 5 --driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native --driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars /opt/spark2/libs/mysql-connector-java-5.1.36.jar " * error messages: 2017-07-17 17:15:25,255 - WARN - org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:78) - rpc-client-1-1 -Exception in connection from /ip:60099 java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel; at org.apache.spark.rpc.netty.NettyRpcHandler.channelActive(NettyRpcEnv.scala:614) at org.apache.spark.network.server.TransportRequestHandler.channelActive(TransportRequestHandler.java:87) at org.apache.spark.network.server.TransportChannelHandler.channelActive(TransportChannelHandler.java:88) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1282) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:887) at org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:262) at org.spark_project.io.netty.channel.nio
Re: Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
Following error we are getting because of dependency mismatch. Regards, vaquar khan On Jul 17, 2017 3:50 AM, "zzcclp" <441586...@qq.com> wrote: Hi guys: I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with following command, error 'NoSuchMethodError: org.apache.spark.network.client.TransportClient. getChannel()Lio/netty/channel/Channel;' appears sometimes: command: *su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master yarn --deploy-mode client --files /opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g --num-executors 2 --executor-memory 8g --executor-cores 5 --driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native --driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars /opt/spark2/libs/mysql-connector-java-5.1.36.jar " * error messages: 2017-07-17 17:15:25,255 - WARN - org.apache.spark.network.server.TransportChannelHandler.exceptionCaught( TransportChannelHandler.java:78) - rpc-client-1-1 -Exception in connection from /ip:60099 java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient. getChannel()Lio/netty/channel/Channel; at org.apache.spark.rpc.netty.NettyRpcHandler.channelActive( NettyRpcEnv.scala:614) at org.apache.spark.network.server.TransportRequestHandler.channelActive( TransportRequestHandler.java:87) at org.apache.spark.network.server.TransportChannelHandler.channelActive( TransportChannelHandler.java:88) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter. channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive( IdleStateHandler.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter. channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter. channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.DefaultChannelPipeline$ HeadContext.channelActive(DefaultChannelPipeline.java:1282) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext. invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive( DefaultChannelPipeline.java:887) at org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe. fulfillConnectPromise(AbstractNioChannel.java:262) at org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe. finishConnect(AbstractNioChannel.java:292) at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKey( NioEventLoop.java:640) at org.spark_project.io.netty.channel.nio.NioEventLoop. processSelectedKeysOptimized(NioEventLoop.java:575) at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeys( NioEventLoop.java:489) at org.spark_project.io.netty.channel.nio.NioEventLoop.run( NioEventLoop.java:451) at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2. run(SingleThreadEventExecutor.java:140) at org.spark_project.io.netty.uti
Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
Hi guys: I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with following command, error 'NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;' appears sometimes: command: *su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master yarn --deploy-mode client --files /opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g --num-executors 2 --executor-memory 8g --executor-cores 5 --driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native --driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars /opt/spark2/libs/mysql-connector-java-5.1.36.jar " * error messages: 2017-07-17 17:15:25,255 - WARN - org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:78) - rpc-client-1-1 -Exception in connection from /ip:60099 java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel; at org.apache.spark.rpc.netty.NettyRpcHandler.channelActive(NettyRpcEnv.scala:614) at org.apache.spark.network.server.TransportRequestHandler.channelActive(TransportRequestHandler.java:87) at org.apache.spark.network.server.TransportChannelHandler.channelActive(TransportChannelHandler.java:88) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198) at org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1282) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205) at org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:887) at org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:262) at org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:292) at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640) at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) at org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) at org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745) 2017-07-17 17:15:25,256 - DEBUG - org
