The default # of partitions is the # of cores, correct?
On 7/18/14, 10:53 AM, Yanbo Liang wrote:
check how many partitions in your program.
If only one, change it to more partitions will make the execution
parallel.
2014-07-18 20:57 GMT+08:00 Madhura <das.madhur...@gmail.com
<mailto:das.madhur...@gmail.com>>:
I am running my program on a spark cluster but when I look into my
UI while
the job is running I see that only one worker does most of the
tasks. My
cluster has one master and 4 workers where the master is also a
worker.
I want my task to complete as quickly as possible and I believe
that if the
number of tasks were to be divided equally among the workers, the
job will
be completed faster.
Is there any way I can customize the umber of job on each worker?
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n10160/Question.png>
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Dividing-tasks-among-Spark-workers-tp10160.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com.
