>From my understanding, it depends on whether you enabled CGroup isolation
or not in Yarn. By default it is not, which means you could allocate one
core but bump a lot of thread in your task to occupy the CPU resource, this
is just a logic limitation. For Yarn CPU isolation you may refer to this
post (
http://hortonworks.com/blog/apache-hadoop-yarn-in-hdp-2-2-isolation-of-cpu-resources-in-your-hadoop-yarn-clusters/
).
Thanks
Jerry
On Tue, Nov 10, 2015 at 9:33 PM, Peter Rudenko <petro.rude...@gmail.com>
wrote:
> Hi i have a question: how does the cores isolation works on spark on yarn.
> E.g. i have a machine with 8 cores, but launched a worker with
> --executor-cores 1, and after doing something like:
>
> rdd.foreachPartition(=>{for all visible cores: burn core in a new tread})
>
> Will it see 1 core or all 8 cores?
>
> Thanks,
> Peter Rudenko
>
>
