Thanks a lot Andrew! Yeah I actually realized that later. I made a silly
mistake here.
On Mon, Oct 20, 2014 at 6:03 PM, Andrew Ash and...@andrewash.com wrote:
Hi Anny, SPARK_WORKER_INSTANCES is the number of copies of spark workers
running on a single box. If you change the number you change how the
hardware you have is split up (useful for breaking large servers into 32GB
heaps each which perform better) but doesn't change the amount of hardware
you have. Because the hardware's the same, you're not going to see huge
performance improvements unless you were in the huge heap scenario.
Typically you should configure the parameters so that SPARK_WORKER_CORES *
SPARK_WORKER_INSTANCES = the number of cores on your machine. If you have
an 8 core box, then you should lower SPARK_WORKER_CORES as you raise
SPARK_WORKER_INSTANCES.
Cheers!
Andrew
On Mon, Oct 20, 2014 at 3:21 PM, anny9699 anny9...@gmail.com wrote:
Hi,
I have a question about the worker_instances setting and worker_cores
setting in aws ec2 cluster. I understand it is a cluster and the default
setting in the cluster is
*SPARK_WORKER_CORES = 8
SPARK_WORKER_INSTANCES = 1*
However after I changed it to
*SPARK_WORKER_CORES = 8
SPARK_WORKER_INSTANCES = 8*
Seems the speed doesn't change very much. Could anyone give an explanation
about this? Maybe more details about work_cores vs worker_instances?
Thanks a lot!
Anny
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/worker-instances-vs-worker-cores-tp16855.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
