OK. Are you sure the executor has the memory you think? "-Xmx24g" in its command line? It may be that for some reason your job is reserving an exceptionally large amount of non-heap memory. I am not sure that's to be expected with the ALS job though. Even if the settings work, considering using the explicit command line configuration.
On Sat, Jan 17, 2015 at 12:49 PM, Antony Mayi <[email protected]> wrote: > the values are for sure applied as expected - confirmed using the spark UI > environment page... > > it comes from my defaults configured using > 'spark.yarn.executor.memoryOverhead=8192' (yes, now increased even more) in > /etc/spark/conf/spark-defaults.conf and 'export SPARK_EXECUTOR_MEMORY=24G' > in /etc/spark/conf/spark-env.sh --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
