Is there more documentation on using spark-submit with Yarn? Trying to launch a simple job does not seem to work.
My run command is as follows: /opt/cloudera/parcels/CDH/bin/spark-submit \ --master yarn \ --deploy-mode client \ --executor-memory 10g \ --driver-memory 10g \ --num-executors 50 \ --class $MAIN_CLASS \ --verbose \ $JAR \ "$@" The verbose logging correctly parses the arguments: System properties: spark.executor.memory -> 10g spark.executor.instances -> 50 SPARK_SUBMIT -> true spark.master -> yarn-client But when I view the job 4040 page, SparkUI, there is a single executor (just the driver node) and I see the following in enviroment spark.master - local[24] Also, when I run with yarn-cluster, how can I access the SparkUI page? Thanks, Arun