Re: spark-submit is ignoring "--executor-cores"

Thanks a lot Saisai and Zhan, I see DefaultResourceCalculator currently being used for Capacity scheduler. We will change it to DominantResourceCalculator. Thanks, Sivakumar Bhavanari. On Mon, Dec 21, 2015 at 5:56 PM, Zhan Zhang wrote: > BTW: It is not only a Yarn-webui

spark-submit is ignoring "--executor-cores"

Hi Everyone, Observing a strange problem while submitting spark streaming job in yarn-cluster mode through spark-submit. All the executors are using only 1 Vcore irrespective value of the parameter --executor-cores. Are there any config parameters overrides --executor-cores value? Thanks,

Re: spark-submit is ignoring "--executor-cores"

Hi Siva, How did you know that --executor-cores is ignored and where did you see that only 1 Vcore is allocated? Thanks Saisai On Tue, Dec 22, 2015 at 9:08 AM, Siva wrote: > Hi Everyone, > > Observing a strange problem while submitting spark streaming job in >

Re: spark-submit is ignoring "--executor-cores"

I guess you're using DefaultResourceCalculator for capacity scheduler, can you please check you capacity scheduler configuration? By default, this resource calculator will only honor memory as resource calculator, so vcores will always show 1 not matter what values you set (but Spark internally

Re: spark-submit is ignoring "--executor-cores"

Hi Saisai, Total Vcores available in yarn applications web UI (runs on 8088) before and after only varies with number of executors + driver core. If I give 10 executors, I see only 11 vcores being used in yarn application web UI. Thanks, Sivakumar Bhavanari. On Mon, Dec 21, 2015 at 5:21 PM,

Re: spark-submit is ignoring "--executor-cores"

BTW: It is not only a Yarn-webui issue. In capacity scheduler, vcore is ignored. If you want Yarn to honor vcore requests, you have to use DominantResourceCalculator as Saisai suggested. Thanks. Zhan Zhang On Dec 21, 2015, at 5:30 PM, Saisai Shao