Hi According to official spec as below, I think the SparkSession.builder gets the highest priority to the configuration and it means the ‘spark-submit’ passing options would be ignored. Please correct me if I am wrong, many thanks.
Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file. https://spark.apache.org/docs/latest/configuration.html Hope this helps, Neo <em...@yeikel.com> 於 2019年1月1日 週二 上午6:04寫道： > Hi Community , > > > > When we submit a job using ‘spark-submit’ passing options like the ‘master > url’ what should be the content of the main class? > > > > For example , if I create the session myself : > > > > val spark = SparkSession.builder. > > master("local[*]") > > .appName("Console") > > .config("spark.app.id", "spark-es") > > .getOrCreate() > > > > Will spark-submit overwrite parameters like the master url , name , etc? > > > > Thank you > [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&;> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&;> 19/01/01 下午3:43:08
