I would have to check the Spark source code, but theoretically you can limit
the no of threads on the jvm level. Maybe spark does this.Alternatively, you
can use cgroups, but this introduces other complexity.
> On 10 Nov 2015, at 14:33, Peter Rudenko <petro.rude...@gmail.com> wrote:
>
> Hi i have a question: how does the cores isolation works on spark on yarn.
> E.g. i have a machine with 8 cores, but launched a worker with
> --executor-cores 1, and after doing something like:
>
> rdd.foreachPartition(=>{for all visible cores: burn core in a new tread})
>
> Will it see 1 core or all 8 cores?
>
> Thanks,
> Peter Rudenko
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
