[
https://issues.apache.org/jira/browse/SPARK-21894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151930#comment-16151930
]
Kazuaki Ishizaki commented on SPARK-21894:
------------------------------------------
Thank you for reporting this issue. Could you please attach a smaller program
that can reproduce this problem?
> Some Netty errors do not propagate to the top level driver
> ----------------------------------------------------------
>
> Key: SPARK-21894
> URL: https://issues.apache.org/jira/browse/SPARK-21894
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Charles Allen
>
> We have an environment with Netty 4.1 (
> https://issues.apache.org/jira/browse/SPARK-19552 for some context) and the
> following error occurs. The reason THIS issue is being filed is because this
> error leaves the Spark workload in a bad state where it does not make any
> progress, and does not shut down.
> The expected behavior is that the spark job would throw an exception that can
> be caught by the driving application.
> {code}
> 017-09-01T16:13:32,175 ERROR [shuffle-server-3-2]
> org.apache.spark.network.server.TransportRequestHandler - Error sending
> result StreamResponse{streamId=/jars/lz4-1.3.0.jar, byteCount=236880,
> body=FileSegmentManagedBuffer{file=/Users/charlesallen/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar,
> offset=0, length=236880}} to /192.168.59.3:56703; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:810)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> ~[netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:194)
> [spark-network-common_2.11-2.1.0-mmx9.jar:2.1.0-mmx9]
> at
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:150)
> [spark-network-common_2.11-2.1.0-mmx9.jar:2.1.0-mmx9]
> at
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:111)
> [spark-network-common_2.11-2.1.0-mmx9.jar:2.1.0-mmx9]
> at
> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119)
> [spark-network-common_2.11-2.1.0-mmx9.jar:2.1.0-mmx9]
> at
> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
> [spark-network-common_2.11-2.1.0-mmx9.jar:2.1.0-mmx9]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
> [spark-network-common_2.11-2.1.0-mmx9.jar:2.1.0-mmx9]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
> [netty-all-4.1.11.Final.jar:4.1.11.Final]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
