Hi all,
I'm doing some testing on a small dataset (HadoopRDD, 2GB, ~10M records), with
a cluster of 3 nodes
Simple calculations like count take approximately 5s when using the default
value of executor.memory (512MB). When I scale this up to 2GB, several Tasks
take 1m or more (while most
Looks like 1 worker is doing the job. Can you repartition the RDD? Also
what is the number of cores that you allocated? Things like this, you can
easily identify by looking at the workers webUI (default worker:8081)
Thanks
Best Regards
On Tue, Aug 19, 2014 at 6:35 PM, Laird, Benjamin
Given a fixed amount of memory allocated to your workers, more memory per
executor means fewer executors can execute in parallel. This means it takes
longer to finish all of the tasks. Set high enough, and your executors can
find no worker with enough memory and so they all are stuck waiting for
, August 19, 2014 at 9:23 AM
To: Capital One
benjamin.la...@capitalone.commailto:benjamin.la...@capitalone.com
Cc: user@spark.apache.orgmailto:user@spark.apache.org
user@spark.apache.orgmailto:user@spark.apache.org
Subject: Re: Executor Memory, Task hangs
Given a fixed amount of memory allocated
