In theory, yes - however in practice it seems that it depends on how they die.
I’ve recently logged an issue for the case when the machine is restarted. If the executor process dies it generally comes back gracefully. https://issues.apache.org/jira/browse/SPARK-10792 Maybe you can vote up the issue if it’s the same use case :) Also – make sure that you have resources available in YARN, if the cluster is shared. -adrian From: Alexander Pivovarov Date: Tuesday, September 29, 2015 at 1:38 AM To: "user@spark.apache.org<mailto:user@spark.apache.org>" Subject: Does YARN start new executor in place of the failed one? Hello Everyone I use Spark on YARN on EMR-4 The spark program which I run has several jobs/stages and run for about 10 hours During the execution some executors might fail for some reason. BUT I do not see that new executor are started in place of the failed ones So, what I see in spark UI is that at the beginning of my program I have 100 executors but in 10 hours I see only 67 executors. I remember that in Standalone mode Spark Worker starts new executor in place of failed one automatically. How to active the same behavior on YARN? The only non-default YARN setting I use are the following: yarn.nodemanager.pmem-check-enabled=false yarn.nodemanager.vmem-check-enabled=false Thank you Alex
