Running on Amazon EMR w/Yarn and Spark 1.1.1, I have trouble getting Yarn
to use the number of executors that I specify in spark-submit:
--num-executors 2
In a cluster with two core nodes will typically only result in one executor
running at a time. I can play with the memory settings and
How many cores / memory do you have available per NodeManager, and how
many cores / memory are you requesting for your job?
Remember that in Yarn mode, Spark launches num executors + 1
containers. The extra container, by default, reserves 1 core and about
1g of memory (more if running in cluster
