Re: SparkConf does not work for spark.driver.memory
On Thu, Feb 18, 2016 at 10:26 AM, wgtmacwrote: > In the code, I did following: > val sc = new SparkContext(new > SparkConf().setAppName("test").set("spark.driver.memory", "4g")) You can't set the driver memory like this, in any deploy mode. When that code runs, the driver is already running, so there's no way to modify the JVM's command line options at that time. -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
SparkConf does not work for spark.driver.memory
Hi I'm using spark 1.5.1. But I encountered a problem using SparkConf to set spark.driver.memory in yarn-cluster mode. Example 1: In the code, I did following: val sc = new SparkContext(new SparkConf().setAppName("test").set("spark.driver.memory", "4g")) And used following command to submit job: YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit \ --class path.to.className \ --jars jarName \ --queue default \ --num-executors 4 \ --conf spark.eventLog.overwrite=true \ --conf spark.eventLog.enabled=true \ --conf spark.eventLog.dir=hdfs://localhost:port \ --conf spark.yarn.historyServer.address=http://host:port Although in the webUI spark.driver.memory is 4g but actually driver memory is 1g (default value) And my job failed due to shortage of driver memory which throws exceeding GC limit exception. Example 2: Almost same configuration except I added "--driver-memory 2g" at the end of the command. Then in the webUI spark.driver.memory is 4g but actually driver memory is 2g now. My job succeeded which proves the driver memory is different with example 1. So my question is that: In yarn-cluster mode, is it ineffective to use SparkConf to set spark.driver.memory? Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkConf-does-not-work-for-spark-driver-memory-tp26270.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org