The problem is clearly to do with the executor exceeding YARN
allocations, so, this can't be in local mode. He said this was running
on YARN at the outset.
On Mon, Jan 19, 2015 at 2:27 AM, Raghavendra Pandey
raghavendra.pan...@gmail.com wrote:
If you are running spark in local mode, executor
If you are running spark in local mode, executor parameters are not used as
there is no executor. You should try to set corresponding driver parameter
to effect it.
On Mon, Jan 19, 2015, 00:21 Sean Owen so...@cloudera.com wrote:
OK. Are you sure the executor has the memory you think? -Xmx24g in
OK. Are you sure the executor has the memory you think? -Xmx24g in
its command line? It may be that for some reason your job is reserving
an exceptionally large amount of non-heap memory. I am not sure that's
to be expected with the ALS job though. Even if the settings work,
considering using the
I'm not sure how you are setting these values though. Where is
spark.yarn.executor.memoryOverhead=6144 ? Env variables aren't the
best way to set configuration either. Again have a look at
http://spark.apache.org/docs/latest/running-on-yarn.html
... --executor-memory 22g --conf
the values are for sure applied as expected - confirmed using the spark UI
environment page...
it comes from my defaults configured using
'spark.yarn.executor.memoryOverhead=8192' (yes, now increased even more) in
/etc/spark/conf/spark-defaults.conf and 'export SPARK_EXECUTOR_MEMORY=24G' in
although this helped to improve it significantly I still run into this problem
despite increasing the spark.yarn.executor.memoryOverhead vastly:
export SPARK_EXECUTOR_MEMORY=24Gspark.yarn.executor.memoryOverhead=6144
yet getting this:2015-01-17 04:47:40,389 WARN
the question really is whether this is expected that the memory requirements
grow rapidly with the rank... as I would expect memory is rather O(1) problem
with dependency only on the size of input data.
if this is expected is there any rough formula to determine the required memory
based on ALS
I would expect the size of the user/item feature RDDs to grow linearly
with the rank, of course. They are cached, so that would drive cache
memory usage on the cluster.
This wouldn't cause executors to fail for running out of memory
though. In fact, your error does not show the task failing for
this seems to have sorted it, awesome, thanks for great help.Antony.
On Sunday, 11 January 2015, 13:02, Sean Owen so...@cloudera.com wrote:
I would expect the size of the user/item feature RDDs to grow linearly
with the rank, of course. They are cached, so that would drive cache
the memory requirements seem to be rapidly growing hen using higher rank... I
am unable to get over 20 without running out of memory. is this
expected?thanks, Antony.
the actual case looks like this:* spark 1.1.0 on yarn (cdh 5.2.1)* ~8-10
executors, 36GB phys RAM per host* input RDD is roughly 3GB containing
~150-200M items (and this RDD is made persistent using .cache())* using pyspark
yarn is configured with the limit yarn.nodemanager.resource.memory-mb of
11 matches
Mail list logo
