YARN will kill your processes if the child processes you start via PIPE consume too much memory, you can configured the amount of memory Spark leaves aside for other processes besides the JVM in the YARN containers with spark.yarn.executor.memoryOverhead.
On Wed, Nov 23, 2016 at 10:38 PM, Sameer Choudhary <sameer2...@gmail.com> wrote: > Hi, > > I am working on an Spark 1.6.2 application on YARN managed EMR cluster > that uses RDD's pipe method to process my data. I start a light weight > daemon process that starts processes for each task via pipes. This is > to ensure that I don't run into > https://issues.apache.org/jira/browse/SPARK-671. > > I'm running into Spark job failure due to task failures across the > cluster. Following are the questions that I think would help in > understanding the issue: > > - How does resource allocation in PySpark work? How does YARN and > SPARK track the memory consumed by python processes launched on the > worker nodes? > > - As an example, let's say SPARK started n tasks on a worker node. > These n tasks start n processes via pipe. Memory for executors is > already reserved during application launch. As the processes run their > memory footprint grows and eventually there is not enough memory on > the box. In this case how will YARN and SPARK behave? Will the > executors be killed or my processes will kill, eventually killing the > task? I think this could lead to cascading failures of tasks across > cluster as retry attempts also fail, eventually leading to termination > of SPARK job. Is there a way to avoid this? > > - When we define number of executors in my SparkConf, are they > distributed evenly across my nodes? One approach to get around this > problem would be to limit the number of executors on each host that > YARN can launch. So we will manage the memory for piped processes > outside of YARN. Is there way to avoid this? > > Thanks, > Sameer > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
