Another aspect to keep in mind is JVM above 8-10GB starts to misbehave.
Typically better to split up ~ 15GB intervals.
if you are choosing machines 10GB/Core is a approx to maintain.
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi https://twitter.com/mayur_rustagi
On Fri, Sep 12, 2014 at 2:59 AM, Sean Owen so...@cloudera.com wrote:
As I understand, there's generally not an advantage to running many
executors per machine. Each will already use all the cores, and
multiple executors just means splitting the available memory instead
of having one big pool. I think there may be an argument at extremes
of scale where one JVM with a huge heap might have excessive GC
pauses, or too many open files, that kind of thing?
On Thu, Sep 11, 2014 at 8:42 PM, Mike Sam mikesam...@gmail.com wrote:
Hi There,
I am new to Spark and I was wondering when you have so much memory on
each
machine of the cluster, is it better to run multiple workers with limited
memory on each machine or is it better to run a single worker with
access to
the majority of the machine memory? If the answer is it depends, would
you
please elaborate?
Thanks,
Mike
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
