Re: Spark on YARN memory consumption

Thanks that explains a lot. -- Jan Sterba https://twitter.com/honzasterba | http://flickr.com/honzasterba | http://500px.com/honzasterba On Fri, Mar 11, 2016 at 2:36 PM, Silvio Fiorito wrote: > Hi Jan, > > > > Yes what you’re seeing is due to YARN container memory

RE: Spark on YARN memory consumption

Hi Jan, Yes what you’re seeing is due to YARN container memory overhead. Also, typically the memory increments for YARN containers is 1GB. This gives a good overview: http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ Thanks, Silvio From: Jan