In the standalone mode, it applies to the Driver JVM processor heap size.
You should consider giving enough memory space to it, in standalone mode, due
to:
1) Any data you bring back to the driver will store in it, like RDD.collect or
DF.show2) The Driver also host a web UI for the application job you are
running, and there could be big memory requirement as huge job related metrics
data, if the job contains lots of stages and tasks.
Yong
> From: carli...@janelia.hhmi.org
> To: user@spark.apache.org
> Subject: spark.driver.memory meaning
> Date: Sun, 3 Apr 2016 14:57:51 +0000
>
> In the spark-env.sh example file, the comments indicate that the
> spark.driver.memory is the memory for the master in YARN mode. None of that
> actually makes any sense…
>
> In any case, I’m using spark in a standalone mode, running the driver on a
> separate machine from the master. I have a few questions regarding that:
>
> Does the spark.driver.memory only work in YARN mode?
>
> Does the value apply to the master or the driver?
>
> If the memory applies to the driver, what is that memory used for?
>
> Does it make sense to change it based on what kind of machine the driver is
> running on? (We have both 256GB nodes and 128GB nodes available for use as
> the driver)
>
> Thanks,
> Ken
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
