Hi,
How to set this parameters while launching spark shell spark.shuffle.memoryFraction=0.5 and spark.yarn.executor.memoryOverhead=1024 I tried giving like this but I am giving below error spark-shell --master yarn --deploy-mode client --driver-memory 16G --num-executors 500 executor-cores 4 --executor-memory 7G --conf spark.shuffle.memoryFraction=0.5 --conf spark.yarn.executor.memoryOverhead=1024 Warning 17/02/13 22:42:02 WARN SparkConf: Detected deprecated memory fraction settings: [spark.shuffle.memoryFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended). On Mon, Feb 13, 2017 at 11:23 PM, Thakrar, Jayesh < jthak...@conversantmedia.com> wrote: > Nancy, > > > > As your log output indicated, your executor 11 GB memory limit. > > While you might want to address the root cause/data volume as suggested by > Jon, you can do an immediate test by changing your command as follows > > > > spark-shell --master yarn --deploy-mode client --driver-memory 16G > --num-executors 500 executor-cores 7 --executor-memory 14G > > > > This essentially increases your executor memory from 11 GB to 14 GB. > > Note that it will result in a potentially large footprint - from 500x11 to > 500x14 GB. > > You may want to consult with your DevOps/Operations/Spark Admin team first. > > > > *From: *Jon Gregg <coble...@gmail.com> > *Date: *Monday, February 13, 2017 at 8:58 AM > *To: *nancy henry <nancyhenry6...@gmail.com> > *Cc: *"user @spark" <user@spark.apache.org> > *Subject: *Re: Lost executor 4 Container killed by YARN for exceeding > memory limits. > > > > Setting Spark's memoryOverhead configuration variable is recommended in > your logs, and has helped me with these issues in the past. Search for > "memoryOverhead" here: http://spark.apache.org/docs/ > latest/running-on-yarn.html > > > > That said, you're running on a huge cluster as it is. If it's possible to > filter your tables down before the join (keeping just the rows/columns you > need), that may be a better solution. > > > > Jon > > > > On Mon, Feb 13, 2017 at 5:27 AM, nancy henry <nancyhenry6...@gmail.com> > wrote: > > Hi All,, > > > > I am getting below error while I am trying to join 3 tables which are in > ORC format in hive from 5 10gb tables through hive context in spark > > > > Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB > physical memory used. Consider boosting spark.yarn.executor. > memoryOverhead. > > 17/02/13 02:21:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: > Container killed by YARN for exceeding memory limits. 11.1 GB of 11 GB > physical memory used > > > > > > I am using below memory parameters to launch shell .. what else i could > increase from these parameters or do I need to change any configuration > settings please let me know > > > > spark-shell --master yarn --deploy-mode client --driver-memory 16G > --num-executors 500 executor-cores 7 --executor-memory 10G > > > > >
