On Mon, Apr 27, 2015 at 7:36 AM, Shuai Zheng szheng.c...@gmail.com wrote:
Thanks. So may I know what is your configuration for more/smaller
executors on r3.8xlarge, how big of the memory that you eventually decide
to give one executor without impact performance (for example: 64g? ).
We're
PM
To: Dean Wampler
Cc: Shuai Zheng; user@spark.apache.org
Subject: Re: Slower performance when bigger memory?
FWIW, I ran into a similar issue on r3.8xlarge nodes and opted for more/smaller
executors. Another observation was that one large executor results in less
overall read throughput from
[mailto:szheng.c...@gmail.com]
Sent: Thursday, April 23, 2015 6:14 PM
To: user@spark.apache.org
Subject: Slower performance when bigger memory?
Hi All,
I am running some benchmark on r3*8xlarge instance. I have a cluster with
one master (no executor on it) and one slave (r3*8xlarge).
My job has
,'cvml','user@spark.apache.org');
*Subject:* Slower performance when bigger memory?
Hi All,
I am running some benchmark on r3*8xlarge instance. I have a cluster with
one master (no executor on it) and one slave (r3*8xlarge).
My job has 1000 tasks in stage 0.
R3*8xlarge has 244G
FWIW, I ran into a similar issue on r3.8xlarge nodes and opted for
more/smaller executors. Another observation was that one large executor
results in less overall read throughput from S3 (using Amazon's EMRFS
implementation) in case that matters to your application.
-Sven
On Thu, Apr 23, 2015 at
Hi All,
I am running some benchmark on r3*8xlarge instance. I have a cluster with
one master (no executor on it) and one slave (r3*8xlarge).
My job has 1000 tasks in stage 0.
R3*8xlarge has 244G memory and 32 cores.
If I create 4 executors, each has 8 core+50G memory, each task
Shuai:
Please take a look at:
http://blog.takipi.com/garbage-collectors-serial-vs-parallel-vs-cms-vs-the-g1-and-whats-new-in-java-8/
On Apr 23, 2015, at 10:18 AM, Dean Wampler deanwamp...@gmail.com wrote:
JVM's often have significant GC overhead with heaps bigger than 64GB. You
might try
