Re: Spark 1.1.0 does not spawn more than 6 executors in yarn-client mode and ignores --num-executors

Hi guys I am hoping someone might have a clue on why this is happening. Otherwise I will have to dwell into YARN module's source code to better understand the issue. On Wed, Dec 10, 2014, 11:54 PM Aniket Bhatnagar aniket.bhatna...@gmail.com wrote: I am running spark 1.1.0 on AWS EMR and I am

Spark 1.1.0 does not spawn more than 6 executors in yarn-client mode and ignores --num-executors

I am running spark 1.1.0 on AWS EMR and I am running a batch job that should seems to be highly parallelizable in yarn-client mode. But spark stop spawning any more executors after spawning 6 executors even though YARN cluster has 15 healthy m1.large nodes. I even tried providing '--num-executors