[ https://issues.apache.org/jira/browse/SPARK-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen reopened SPARK-4299: ------------------------------ Wait a sec. I am not clear it's resolved on testing with 1.3.0. But, it is a duplicate of SPARK-3884 > In spark-submit, the driver-memory value is used for the > SPARK_SUBMIT_DRIVER_MEMORY value > ----------------------------------------------------------------------------------------- > > Key: SPARK-4299 > URL: https://issues.apache.org/jira/browse/SPARK-4299 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.1.0 > Reporter: Virgile Devaux > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > In the spark-submit script, the lines below: > elif [ "$1" = "--driver-memory" ]; then > export SPARK_SUBMIT_DRIVER_MEMORY=$2 > are wrong: spark-submit is not the process that will handle the driver when > you're in yarn-cluster mode. So, when I lanch spark-submit on a light server > with only 2Gb of memory and want to allocate 4gb of memory to the driver > (that will run in the ressource manager on a big fat yarn server with, say, > 64Gb of RAM) spark submit fails with a OutOfMemory. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org