Thanks that explains a lot. -- Jan Sterba https://twitter.com/honzasterba | http://flickr.com/honzasterba | http://500px.com/honzasterba
On Fri, Mar 11, 2016 at 2:36 PM, Silvio Fiorito <silvio.fior...@granturing.com> wrote: > Hi Jan, > > > > Yes what you’re seeing is due to YARN container memory overhead. Also, > typically the memory increments for YARN containers is 1GB. > > > > This gives a good overview: > http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ > > > > Thanks, > > Silvio > > > > > > > > From: Jan Štěrba > Sent: Friday, March 11, 2016 8:27 AM > To: User > Subject: Spark on YARN memory consumption > > > > Hello, > > I am exprimenting with tuning an on demand spark-cluster on top of our > cloudera hadoop. I am running Cloudera 5.5.2 with Spark 1.5 right now > and I am running spark in yarn-client mode. > > Right now my main experimentation is about spark.executor.memory > property and I have noticed a strange behaviour. > > When I set spark.executor.memory=512M several things happen: > - per each executor a container with 1GB memory is requested and > assigned from YARN > - in Spark UI I can see that each executor has 256M memory > > So what I am seeing is that spark requests 2x the memory but the > executor has only 1/4 of what has been requested. Why is that? > > Thanks. > > -- > Jan Sterba > https://twitter.com/honzasterba | http://flickr.com/honzasterba | > http://500px.com/honzasterba > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
