Hi Xiangrui,
(2014/06/18 6:03), Xiangrui Meng wrote:
Are you using Spark 1.0 or 0.9? Could you go to the executor tab of
the web UI and check the driver's memory?
I am using Spark 1.0.
588.8 MB is allocated for <driver> RDDs.
I am setting SPARK_DRIVER_MEMORY=2g in the conf/spark-env.sh.
The value allocated for <driver> RDDs in the web UI was not changed by
doing as follows:
$ SPARK_DRIVER_MEMORY=6g bin/spark-shell
I set "-verbose:gc" but full GC (or continuous GCs) does not happen
during the aggregate at the driver.
Thanks,
Makoto
