KaiXu created SPARK-27289:
-----------------------------
Summary: spark-submit explicit configuration does not take effect
but Spark UI shows it's effective
Key: SPARK-27289
URL: https://issues.apache.org/jira/browse/SPARK-27289
Project: Spark
Issue Type: Bug
Components: Deploy, Documentation, Spark Submit, Web UI
Affects Versions: 2.3.3
Reporter: KaiXu
The [doc
|https://spark.apache.org/docs/latest/submitting-applications.html]says that
"In general, configuration values explicitly set on a {{SparkConf}} take the
highest precedence, then flags passed to {{spark-submit}}, then values in the
defaults file", but when setting spark.local.dir through --conf with
spark-submit, it still uses the values from
${SPARK_HOME}/conf/spark-defaults.conf, what's more, the Spark runtime UI
environment variables shows the value from --conf, which is really misleading.
e.g.
I set submit my application through the command:
/opt/spark233/bin/spark-submit --properties-file /opt/spark.conf --conf
spark.local.dir=/tmp/spark_local -v --class
org.apache.spark.examples.mllib.SparseNaiveBayes --master
spark://bdw-slave20:7077
/opt/sparkbench/assembly/target/sparkbench-assembly-7.1-SNAPSHOT-dist.jar
hdfs://bdw-slave20:8020/Bayes/Input
the spark.local.dir in ${SPARK_HOME}/conf/spark-defaults.conf is:
spark.local.dir=/mnt/nvme1/spark_local
when the application is running, I found the intermediate shuffle data was
wrote to /mnt/nvme1/spark_local, which is set through
${SPARK_HOME}/conf/spark-defaults.conf, but the Web UI shows that the
environment value spark.local.dir=/tmp/spark_local.
The spark-submit verbose also shows spark.local.dir=/tmp/spark_local, it's
misleading.
!image-2019-03-27-10-59-38-377.png!
spark-submit verbose:
XXXX
Spark properties used, including those specified through
--conf and those from the properties file /opt/spark.conf:
(spark.local.dir,/tmp/spark_local)
(spark.default.parallelism,132)
(spark.driver.memory,10g)
(spark.executor.memory,352g)
XXXXX
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
