Hi Imran,

Thanks for pointing that out. My data comes from the HBase connector of
Spark. I do not govern the distribution of data myself. HBase decides to
put the data on any of the region servers. Is there a way to distribute
data evenly? And I am especially interested in running even small loads
very quickly apart from bulk loads.

Thanks,
Jatin

On Thu, Feb 19, 2015 at 10:28 PM, Imran Rashid <[email protected]> wrote:

> almost all your data is going to one task.  You can see that the shuffle
> read for task 0 is 153.3 KB, and for most other tasks its just 26B (which
> is probably just some header saying there are no actual records).  You need
> to ensure your data is more evenly distributed before this step.
>
> On Thu, Feb 19, 2015 at 10:53 AM, jatinpreet <[email protected]> wrote:
>
>> Hi,
>>
>> I am running Spark 1.2.1 for compute intensive jobs comprising of multiple
>> tasks. I have observed that most tasks complete very quickly, but there
>> are
>> always one or two tasks that take a lot of time to complete thereby
>> increasing the overall stage time. What could be the reason for this?
>>
>> Following are the statistics for one such stage. As you can see, the task
>> with index 0 takes 1.1 minutes whereas others completed much more quickly.
>>
>> Aggregated Metrics by Executor
>> Executor ID     Address         Task Time       Total Tasks     Failed
>> Tasks    Succeeded Tasks
>> Input   Output  Shuffle Read    Shuffle Write   Shuffle Spill (Memory)
>> Shuffle
>> Spill (Disk)
>> 0       slave1:56311    46 s    13      0       13      0.0 B   0.0 B
>>  0.0 B   0.0 B   0.0 B   0.0 B
>> 1       slave2:42648    2.1 min         13      0       13      0.0 B
>>  0.0 B   384.3 KB        0.0 B   0.0 B
>> 0.0 B
>> 2       slave3:44322    23 s    12      0       12      0.0 B   0.0 B
>>  136.4 KB        0.0 B   0.0 B   0.0
>> B
>> 3       slave4:37987    44 s    12      0       12      0.0 B   0.0 B
>>  213.9 KB        0.0 B   0.0 B   0.0
>> B
>> Tasks
>> Index   ID      Attempt Status  Locality Level  Executor ID / Host
>> Launch Time
>> Duration        GC Time Shuffle Read    Errors
>> 0       213     0       SUCCESS         PROCESS_LOCAL   1 / slave2
>> 2015/02/19 11:40:05     1.1 min
>> 1 s     153.3 KB
>> 5       218     0       SUCCESS         PROCESS_LOCAL   3 / slave4
>> 2015/02/19 11:40:05     23 ms
>> 26.0 B
>> 1       214     0       SUCCESS         PROCESS_LOCAL   3 / slave4
>> 2015/02/19 11:40:05     2 s     0.9
>> s       13.8 KB
>> 4       217     0       SUCCESS         PROCESS_LOCAL   1 / slave2
>> 2015/02/19 11:40:05     26 ms
>> 26.0 B
>> 3       216     0       SUCCESS         PROCESS_LOCAL   0 / slave1
>> 2015/02/19 11:40:05     11 ms
>> 0.0 B
>> 2       215     0       SUCCESS         PROCESS_LOCAL   2 / slave3
>> 2015/02/19 11:40:05     27 ms
>> 26.0 B
>> 7       220     0       SUCCESS         PROCESS_LOCAL   0 / slave1
>> 2015/02/19 11:40:05     11 ms
>> 0.0 B
>> 10      223     0       SUCCESS         PROCESS_LOCAL   2 / slave3
>> 2015/02/19 11:40:05     23 ms
>> 26.0 B
>> 6       219     0       SUCCESS         PROCESS_LOCAL   2 / slave3
>> 2015/02/19 11:40:05     23 ms
>> 26.0 B
>> 9       222     0       SUCCESS         PROCESS_LOCAL   3 / slave4
>> 2015/02/19 11:40:05     23 ms
>> 26.0 B
>> 8       221     0       SUCCESS         PROCESS_LOCAL   1 / slave2
>> 2015/02/19 11:40:05     23 ms
>> 26.0 B
>> 11      224     0       SUCCESS         PROCESS_LOCAL   0 / slave1
>> 2015/02/19 11:40:05     10 ms
>> 0.0 B
>> 14      227     0       SUCCESS         PROCESS_LOCAL   2 / slave3
>> 2015/02/19 11:40:05     24 ms
>> 26.0 B
>> 13      226     0       SUCCESS         PROCESS_LOCAL   3 / slave4
>> 2015/02/19 11:40:05     23 ms
>> 26.0 B
>> 16      229     0       SUCCESS         PROCESS_LOCAL   1 / slave2
>> 2015/02/19 11:40:05     22 ms
>> 26.0 B
>> 12      225     0       SUCCESS         PROCESS_LOCAL   1 / slave2
>> 2015/02/19 11:40:05     22 ms
>> 26.0 B
>> 15      228     0       SUCCESS         PROCESS_LOCAL   0 / slave1
>> 2015/02/19 11:40:05     10 ms
>> 0.0 B
>> 17      230     0       SUCCESS         PROCESS_LOCAL   3 / slave4
>> 2015/02/19 11:40:05     22 ms
>> 26.0 B
>> 23      236     0       SUCCESS         PROCESS_LOCAL   0 / slave1
>> 2015/02/19 11:40:05     10 ms
>> 0.0 B
>> 22      235     0       SUCCESS         PROCESS_LOCAL   2 / slave3
>> 2015/02/19 11:40:05     21 ms
>> 26.0 B
>> 19      232     0       SUCCESS         PROCESS_LOCAL   0 / slave1
>> 2015/02/19 11:40:05     10 ms
>> 0.0 B
>> 21      234     0       SUCCESS         PROCESS_LOCAL   3 / slave4
>> 2015/02/19 11:40:05     25 ms
>> 26.0 B
>> 18      231     0       SUCCESS         PROCESS_LOCAL   2 / slave3
>> 2015/02/19 11:40:05     24 ms
>> 26.0 B
>> 20      233     0       SUCCESS         PROCESS_LOCAL   1 / slave2
>> 2015/02/19 11:40:05     28 ms
>> 26.0 B
>> 25      238     0       SUCCESS         PROCESS_LOCAL   3 / slave4
>> 2015/02/19 11:40:05     20 ms
>> 26.0 B
>> 28      241     0       SUCCESS         PROCESS_LOCAL   1 / slave2
>> 2015/02/19 11:40:05     27 ms
>> 26.0 B
>> 27      240     0       SUCCESS         PROCESS_LOCAL   0 / slave1
>> 2015/02/19 11:40:05     10 ms
>> 0.0 B
>>
>>
>> Thanks
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Some-tasks-taking-too-much-time-to-complete-in-a-stage-tp21724.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>
>


-- 
Regards,
Jatinpreet Singh

Reply via email to