Hi
I'm using spark 1.5.1. But I encountered a problem using SparkConf to set
spark.driver.memory in yarn-cluster mode.
Example 1:
In the code, I did following:
val sc = new SparkContext(new
SparkConf().setAppName("test").set("spark.driver.memory", "4g"))
And used following command to submit job:
YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit \
--class path.to.className \
--jars jarName \
--queue default \
--num-executors 4 \
--conf spark.eventLog.overwrite=true \
--conf spark.eventLog.enabled=true \
--conf spark.eventLog.dir=hdfs://localhost:port \
--conf spark.yarn.historyServer.address=http://host:port
Although in the webUI spark.driver.memory is 4g but actually driver memory
is 1g (default value)
And my job failed due to shortage of driver memory which throws exceeding GC
limit exception.
Example 2:
Almost same configuration except I added "--driver-memory 2g" at the end of
the command.
Then in the webUI spark.driver.memory is 4g but actually driver memory is 2g
now.
My job succeeded which proves the driver memory is different with example 1.
So my question is that: In yarn-cluster mode, is it ineffective to use
SparkConf to set spark.driver.memory?
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SparkConf-does-not-work-for-spark-driver-memory-tp26270.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
