Nancy, As the message from Spark indicates, spark.shuffle.memoryFraction is no longer used. It’s a unified heap space for both data caching and other things. Also, the previous 11 GB was not sufficient, and you are making the executor memory even smaller, not sure how it will work.
From: nancy henry <nancyhenry6...@gmail.com> Date: Tuesday, February 14, 2017 at 1:04 AM To: Conversant <jthak...@conversantmedia.com> Cc: Jon Gregg <coble...@gmail.com>, "user @spark" <user@spark.apache.org> Subject: Re: Lost executor 4 Container killed by YARN for exceeding memory limits. Hi, How to set this parameters while launching spark shell spark.shuffle.memoryFraction=0.5 and spark.yarn.executor.memoryOverhead=1024 I tried giving like this but I am giving below error spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 4 --executor-memory 7G --conf spark.shuffle.memoryFraction=0.5 --conf spark.yarn.executor.memoryOverhead=1024 Warning 17/02/13 22:42:02 WARN SparkConf: Detected deprecated memory fraction settings: [spark.shuffle.memoryFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended). On Mon, Feb 13, 2017 at 11:23 PM, Thakrar, Jayesh <jthak...@conversantmedia.com<mailto:jthak...@conversantmedia.com>> wrote: Nancy, As your log output indicated, your executor 11 GB memory limit. While you might want to address the root cause/data volume as suggested by Jon, you can do an immediate test by changing your command as follows spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 7 --executor-memory 14G This essentially increases your executor memory from 11 GB to 14 GB. Note that it will result in a potentially large footprint - from 500x11 to 500x14 GB. You may want to consult with your DevOps/Operations/Spark Admin team first. From: Jon Gregg <coble...@gmail.com<mailto:coble...@gmail.com>> Date: Monday, February 13, 2017 at 8:58 AM To: nancy henry <nancyhenry6...@gmail.com<mailto:nancyhenry6...@gmail.com>> Cc: "user @spark" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Lost executor 4 Container killed by YARN for exceeding memory limits. Setting Spark's memoryOverhead configuration variable is recommended in your logs, and has helped me with these issues in the past. Search for "memoryOverhead" here: http://spark.apache.org/docs/latest/running-on-yarn.html That said, you're running on a huge cluster as it is. If it's possible to filter your tables down before the join (keeping just the rows/columns you need), that may be a better solution. Jon On Mon, Feb 13, 2017 at 5:27 AM, nancy henry <nancyhenry6...@gmail.com<mailto:nancyhenry6...@gmail.com>> wrote: Hi All,, I am getting below error while I am trying to join 3 tables which are in ORC format in hive from 5 10gb tables through hive context in spark Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. 17/02/13 02:21:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB physical memory used I am using below memory parameters to launch shell .. what else i could increase from these parameters or do I need to change any configuration settings please let me know spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 7 --executor-memory 10G
