It's not possible to specify YARN RM paramers at command line of
spark-submit time. You have to specify all resources that are available on
your cluster to YARN upfront. If you want to limit amount of resource
available for your Spark job, consider using YARN dynamic resource pools
instead
Hi,
I'd like to specify the total sum of cores / memory as command line
arguments with spark-submit. That is, I'd like to set
yarn.nodemanager.resource.memory-mb and the
yarn.nodemanager.resource.cpu-vcores parameters as described in this
blog
