Jean-Baptiste Onofré created SPARK-11065:
--------------------------------------------
Summary: IOException thrown at job submit shutdown
Key: SPARK-11065
URL: https://issues.apache.org/jira/browse/SPARK-11065
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 1.6.0
Reporter: Jean-Baptiste Onofré
When submitted a job (for instance JavaWordCount example), even if the job
works fine, at the end of execution, we can see:
{code}
checkForCorruptJournalFiles="true": 1
15/10/12 16:31:12 INFO SparkUI: Stopped Spark web UI at
http://192.168.134.10:4040
15/10/12 16:31:12 INFO DAGScheduler: Stopping DAGScheduler
15/10/12 16:31:12 INFO SparkDeploySchedulerBackend: Shutting down all executors
15/10/12 16:31:12 INFO SparkDeploySchedulerBackend: Asking each executor to
shut down
15/10/12 16:31:12 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
15/10/12 16:31:12 INFO MemoryStore: MemoryStore cleared
15/10/12 16:31:12 INFO BlockManager: BlockManager stopped
15/10/12 16:31:12 INFO BlockManagerMaster: BlockManagerMaster stopped
15/10/12 16:31:12 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
15/10/12 16:31:12 ERROR TransportResponseHandler: Still have 1 requests
outstanding when connection from localhost/127.0.0.1:7077 is closed
15/10/12 16:31:12 ERROR NettyRpcEnv: Exception when sending
RequestMessage(192.168.134.10:40548,NettyRpcEndpointRef(spark://Master@localhost:7077),UnregisterApplication(app-20151012163109-0000),false)
java.io.IOException: Connection from localhost/127.0.0.1:7077 closed
at
org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:104)
at
org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:91)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
at
io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739)
at
io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
15/10/12 16:31:12 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down
remote daemon.
15/10/12 16:31:12 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon
shut down; proceeding with flushing remote transports.
15/10/12 16:31:12 INFO SparkContext: Successfully stopped SparkContext
15/10/12 16:31:12 INFO ShutdownHookManager: Shutdown hook called
15/10/12 16:31:12 INFO ShutdownHookManager: Deleting directory
/tmp/spark-81bc4324-1268-4e54-bdd2-f7a2a36dafd4
{code}
I gonna investigate about that and I will submit a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
