Try to set --total-executor-cores to limit how many total cores it can use.
Thanks Regards,
Meethu M
On Thursday, 2 October 2014 2:39 AM, Akshat Aranya aara...@gmail.com wrote:
I guess one way to do so would be to run 1 worker per node, like say, instead
of running 1 worker and giving
Hi,
What's the relationship between Spark worker and executor memory settings
in standalone mode? Do they work independently or does the worker cap
executor memory?
Also, is the number of concurrent executors per worker capped by the number
of CPU cores configured for the worker?
On Wed, Oct 1, 2014 at 11:33 AM, Akshat Aranya aara...@gmail.com wrote:
On Wed, Oct 1, 2014 at 11:00 AM, Boromir Widas vcsub...@gmail.com wrote:
1. worker memory caps executor.
2. With default config, every job gets one executor per worker. This
executor runs with all cores available to
One indirect way to control the number of cores used in an executor is to
set spark.cores.max and set spark.deploy.spreadOut to be true. The
scheduler in the standalone cluster then assigns roughly the same number of
cores (spark.cores.max/number of worker nodes) to each executor for an
