Re: spark.driver.memory meaning

[Carlile, Ken]

Cool. My users tend to interact with the driver via iPython Notebook, so clearly I’ll have to leave (fairly significant amounts of) ram for that. But I should be able to write a one liner into the spark-env.sh that will determine whether it’s on a 128 or 256GB node and have it size itself accordingly.  Thanks! —Ken On Apr 3, 2016, at 11:06 AM, Yong Zhang <java8...@hotmail.com> wrote: In the standalone mode, it applies to the Driver JVM processor heap size. You should consider giving enough memory space to it, in standalone mode, due to: 1) Any data you bring back to the driver will store in it, like RDD.collect or DF.show 2) The Driver also host a web UI for the application job you are running, and there could be big memory requirement as huge job related metrics data, if the job contains lots of stages and tasks. Yong > From: carli...@janelia.hhmi.org > To: user@spark.apache.org > Subject: spark.driver.memory meaning > Date: Sun, 3 Apr 2016 14:57:51 +0000 >  > In the spark-env.sh example file, the comments indicate that the spark.driver.memory is the memory for the master in YARN mode. None of that actually makes any sense…  >  > In any case, I’m using spark in a standalone mode, running the driver on a separate machine from the master. I have a few questions regarding that:  >  > Does the spark.driver.memory only work in YARN mode?  >  > Does the value apply to the master or the driver?  >  > If the memory applies to the driver, what is that memory used for?  >  > Does it make sense to change it based on what kind of machine the driver is running on? (We have both 256GB nodes and 128GB nodes available for use as the driver) >  > Thanks, > Ken > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >  --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org