There are two related options: To solve your problem directly try:
val conf = new SparkConf().set("spark.yarn.driver.memoryOverhead", "1024") val sc = new SparkContext(conf) And the second, which increases the overall memory available on the driver, as part of your spark-submit script add: --driver-memory 2g Hope this helps! From: David McWhorter <mcwhor...@ccri.com<mailto:mcwhor...@ccri.com>> Date: Monday, January 12, 2015 at 11:01 AM To: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: configuring spark.yarn.driver.memoryOverhead on Spark 1.2.0 Hi all, I'm trying to figure out how to set this option: " spark.yarn.driver.memoryOverhead" on Spark 1.2.0. I found this helpful overview http://apache-spark-user-list.1001560.n3.nabble.com/Stable-spark-streaming-app-td14105.html#a14476, which suggests to launch with --spark.yarn.driver.memoryOverhead 1024 added to spark-submit. However, when I do that I get this error: Error: Unrecognized option '--spark.yarn.driver.memoryOverhead'. Run with --help for usage help or --verbose for debug output I have also tried calling sparkConf.set("spark.yarn.driver.memoryOverhead", "1024") on my spark configuration object but I still get "Will allocate AM container, with XXXX MB memory including 384 MB overhead" when launching. I'm running in yarn-cluster mode. Any help or tips would be appreciated. Thanks, David -- David McWhorter Software Engineer Commonwealth Computer Research, Inc. 1422 Sachem Place, Unit #1 Charlottesville, VA 22901 mcwhor...@ccri.com<mailto:mcwhor...@ccri.com> | 434.299.0090x204 ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.