Hi,
I'd like to specify the total sum of cores / memory as command line
arguments with spark-submit. That is, I'd like to set
yarn.nodemanager.resource.memory-mb and the
yarn.nodemanager.resource.cpu-vcores parameters as described in this
blog
<http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/>
post.
when submitting through the command line, what is the correct way to do
it? Is it:
--conf spark.yarn.nodemanager.resource.memory-mb=54g
--conf spark.yarn.nodemanager.resource.cpu-vcores=31
or
--conf yarn.nodemanager.resource.memory-mb=54g
--conf yarn.nodemanager.resource.cpu-vcores=31
or something else? I tried these, and I tried looking in the
ResourceManager UI to see if they were set, but couldn't find them.
Thanks!
Alexander