Saisai, thanks a ton :)
Regards, Gourav On Mon, Sep 11, 2017 at 11:36 PM, Xiaoye Sun <sunxiaoy...@gmail.com> wrote: > Hi Jerry, > > This solves my problem. 🙏 thanks > > On Sun, Sep 10, 2017 at 8:19 PM Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >> I guess you're using Capacity Scheduler with DefaultResourceCalculator, >> which doesn't count cpu cores into resource calculation, this "1" you saw >> is actually meaningless. If you want to also calculate cpu resource, you >> should choose DominantResourceCalculator. >> >> Thanks >> Jerry >> >> On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <sunxiaoy...@gmail.com> wrote: >> >>> Hi, >>> >>> I am using Spark 1.6.1 and Yarn 2.7.4. >>> I want to submit a Spark application to a Yarn cluster. However, I found >>> that the number of vcores assigned to a container/executor is always 1, >>> even if I set spark.executor.cores=2. I also found the number of tasks an >>> executor runs concurrently is 2. So, it seems that Spark knows that an >>> executor/container has two CPU cores but the request is not correctly sent >>> to Yarn resource scheduler. I am using the org.apache.hadoop.yarn. >>> server.resourcemanager.scheduler.capacity.CapacityScheduler on Yarn. >>> >>> I am wondering that is it possible to assign multiple vcores to a >>> container when a Spark job is submitted to a Yarn cluster in yarn-cluster >>> mode. >>> >>> Thanks! >>> Best, >>> Xiaoye >>> >> >>
