Hello Everyone I use Spark on YARN on EMR-4
The spark program which I run has several jobs/stages and run for about 10 hours During the execution some executors might fail for some reason. BUT I do not see that new executor are started in place of the failed ones So, what I see in spark UI is that at the beginning of my program I have 100 executors but in 10 hours I see only 67 executors. I remember that in Standalone mode Spark Worker starts new executor in place of failed one automatically. How to active the same behavior on YARN? The only non-default YARN setting I use are the following: yarn.nodemanager.pmem-check-enabled=false yarn.nodemanager.vmem-check-enabled=false Thank you Alex
