Greetings, I am using Spark 2.0.2 with Scala 2.11.7 and Hadoop 2.7.3. When I run spark-submit local mode, I get a netty exception like the following. The code runs fine with Spark 1.6.3, Scala 2.10.x and Hadoop 2.7.3.
6/11/24 08:18:24 ERROR server.TransportRequestHandler: Error sending result StreamResponse{streamId=/jars/simple-project_2.11-1.0.jar, byteCount=3662, body=FileSegmentManagedBuffer{file=/home/hdadmin/Examples/ spark/wordcount/target/scala-2.11/simple-project_2.11-1.0.jar, offset=0, length=3662}} to /10.0.2.15:33926; closing connection io.netty.handler.codec.EncoderException: java.lang.NoSuchMethodError: io.netty.channel.DefaultFileRegion.<init>(Ljava/io/File;JJ)V at io.netty.handler.codec.MessageToMessageEncoder.write( MessageToMessageEncoder.java:107) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite( AbstractChannelHandlerContext.java:658) at io.netty.channel.AbstractChannelHandlerContext.write( AbstractChannelHandlerContext.java:716) at io.netty.channel.AbstractChannelHandlerContext.write( AbstractChannelHandlerContext.java:651) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:266) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite( AbstractChannelHandlerContext.java:658) at io.netty.channel.AbstractChannelHandlerContext.write( AbstractChannelHandlerContext.java:716) at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush( AbstractChannelHandlerContext.java:706) at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush( AbstractChannelHandlerContext.java:741) at io.netty.channel.DefaultChannelPipeline.writeAndFlush( DefaultChannelPipeline.java:895) at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240) at org.apache.spark.network.server.TransportRequestHandler.respond( TransportRequestHandler.java:194) at org.apache.spark.network.server.TransportRequestHandler. processStreamRequest(TransportRequestHandler.java:150) at org.apache.spark.network.server.TransportRequestHandler.handle( TransportRequestHandler.java:111) at org.apache.spark.network.server.TransportChannelHandler.channelRead0( TransportChannelHandler.java:119) at org.apache.spark.network.server.TransportChannelHandler.channelRead0( TransportChannelHandler.java:51) at io.netty.channel.SimpleChannelInboundHandler.channelRead( SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( AbstractChannelHandlerContext.java:319) at io.netty.handler.timeout.IdleStateHandler.channelRead( IdleStateHandler.java:254) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.MessageToMessageDecoder.channelRead( MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( AbstractChannelHandlerContext.java:319) at org.apache.spark.network.util.TransportFrameDecoder.channelRead( TransportFrameDecoder.java:85) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead( DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2. run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoSuchMethodError: io.netty.channel.DefaultFileRegion.<init>(Ljava/io/File;JJ)V at org.apache.spark.network.buffer.FileSegmentManagedBuffer.convertToNetty( FileSegmentManagedBuffer.java:133) at org.apache.spark.network.protocol.MessageEncoder. encode(MessageEncoder.java:54) at org.apache.spark.network.protocol.MessageEncoder. encode(MessageEncoder.java:33) at io.netty.handler.codec.MessageToMessageEncoder.write( MessageToMessageEncoder.java:89) ... 35 more 16/11/24 08:18:24 ERROR client.TransportResponseHandler: Still have 1 requests outstanding when connection from /10.0.2.15:54561 is closed PLEASE ADVISE. Sincerely, Karthik