It's not possible to specify YARN RM paramers at command line of spark-submit time. You have to specify all resources that are available on your cluster to YARN upfront. If you want to limit amount of resource available for your Spark job, consider using YARN dynamic resource pools instead
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v5-1-x/Cloudera-Manager-Managing-Clusters/cm5mc_resource_pools.html -- Ruslan Dautkhanov On Thu, Jul 2, 2015 at 4:20 PM, Alexander Waldin <awal...@inflection.com> wrote: > Hi, > > I'd like to specify the total sum of cores / memory as command line > arguments with spark-submit. That is, I'd like to set > yarn.nodemanager.resource.memory-mb and the > yarn.nodemanager.resource.cpu-vcores parameters as described in this blog > <http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/> > post. > > when submitting through the command line, what is the correct way to do > it? Is it: > > --conf spark.yarn.nodemanager.resource.memory-mb=54g > --conf spark.yarn.nodemanager.resource.cpu-vcores=31 > > or > > --conf yarn.nodemanager.resource.memory-mb=54g > --conf yarn.nodemanager.resource.cpu-vcores=31 > > > or something else? I tried these, and I tried looking in the > ResourceManager UI to see if they were set, but couldn't find them. > > Thanks! > > Alexander >
