[ https://issues.apache.org/jira/browse/SPARK-14611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-14611. ------------------------------------ Resolution: Not A Problem Yes, that's what re-attempts are for. If you don't want a re-attempt, then you can disable it in the configuration. If the re-attempt fails, then it will try again up to the configured limit. > Second attempt observed after AM fails due to max number of executor failure > in first attempt > --------------------------------------------------------------------------------------------- > > Key: SPARK-14611 > URL: https://issues.apache.org/jira/browse/SPARK-14611 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.6.1 > Environment: RHEL7 64 bit > Reporter: Kshitij Badani > > I submitted a spark application in yarn-cluster mode. My cluster has two > Nodemanagers. After submitting the spark application, I tried to restart > Nodemanager on node1 actively running a few executor and this node was not > running the AM. > During the time when the Nodemanager was restarting, 3 of the executors > running on node2 failed with 'failed to connect to external shuffle server' > as follows > java.io.IOException: Failed to connect to node1 > at > org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216) > at > org.apache.spark.network.client.TransportClientFactory.createUnmanagedClient(TransportClientFactory.java:181) > at > org.apache.spark.network.shuffle.ExternalShuffleClient.registerWithShuffleServer(ExternalShuffleClient.java:141) > at > org.apache.spark.storage.BlockManager$$anonfun$registerWithExternalShuffleServer$1.apply$mcVI$sp(BlockManager.scala:211) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) > at > org.apache.spark.storage.BlockManager.registerWithExternalShuffleServer(BlockManager.scala:208) > at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194) > at org.apache.spark.executor.Executor.<init>(Executor.scala:86) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:83) > at > org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116) > at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204) > at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100) > at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.net.ConnectException: Connection refused: node1 > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > at > io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) > at > io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > Each of the 3 executors tried to connect to external shuffle service 2 more > times, all during the period when the NM on node1 was restarting and > eventually failed > Since 3 executors failed, the AM exitted with FAILURE status and I can see > following message in the application logs > INFO ApplicationMaster: Final app status: FAILED, exitCode: 11, (reason: Max > number of executor failures (3) reached) > After this, we saw a 2nd application attempt which succeeded as the NM had > came up back. > Should we see a 2nd attempt in such scenarios where multiple executors have > failed in the 1st attempt due to not being able to connect to external > shuffle service? What if the 2nd attempt also fails due to similar reason, in > that case it would be a heavy penalty? -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org