Hi Imran, Thanks for pointing that out. My data comes from the HBase connector of Spark. I do not govern the distribution of data myself. HBase decides to put the data on any of the region servers. Is there a way to distribute data evenly? And I am especially interested in running even small loads very quickly apart from bulk loads.
Thanks, Jatin On Thu, Feb 19, 2015 at 10:28 PM, Imran Rashid <[email protected]> wrote: > almost all your data is going to one task. You can see that the shuffle > read for task 0 is 153.3 KB, and for most other tasks its just 26B (which > is probably just some header saying there are no actual records). You need > to ensure your data is more evenly distributed before this step. > > On Thu, Feb 19, 2015 at 10:53 AM, jatinpreet <[email protected]> wrote: > >> Hi, >> >> I am running Spark 1.2.1 for compute intensive jobs comprising of multiple >> tasks. I have observed that most tasks complete very quickly, but there >> are >> always one or two tasks that take a lot of time to complete thereby >> increasing the overall stage time. What could be the reason for this? >> >> Following are the statistics for one such stage. As you can see, the task >> with index 0 takes 1.1 minutes whereas others completed much more quickly. >> >> Aggregated Metrics by Executor >> Executor ID Address Task Time Total Tasks Failed >> Tasks Succeeded Tasks >> Input Output Shuffle Read Shuffle Write Shuffle Spill (Memory) >> Shuffle >> Spill (Disk) >> 0 slave1:56311 46 s 13 0 13 0.0 B 0.0 B >> 0.0 B 0.0 B 0.0 B 0.0 B >> 1 slave2:42648 2.1 min 13 0 13 0.0 B >> 0.0 B 384.3 KB 0.0 B 0.0 B >> 0.0 B >> 2 slave3:44322 23 s 12 0 12 0.0 B 0.0 B >> 136.4 KB 0.0 B 0.0 B 0.0 >> B >> 3 slave4:37987 44 s 12 0 12 0.0 B 0.0 B >> 213.9 KB 0.0 B 0.0 B 0.0 >> B >> Tasks >> Index ID Attempt Status Locality Level Executor ID / Host >> Launch Time >> Duration GC Time Shuffle Read Errors >> 0 213 0 SUCCESS PROCESS_LOCAL 1 / slave2 >> 2015/02/19 11:40:05 1.1 min >> 1 s 153.3 KB >> 5 218 0 SUCCESS PROCESS_LOCAL 3 / slave4 >> 2015/02/19 11:40:05 23 ms >> 26.0 B >> 1 214 0 SUCCESS PROCESS_LOCAL 3 / slave4 >> 2015/02/19 11:40:05 2 s 0.9 >> s 13.8 KB >> 4 217 0 SUCCESS PROCESS_LOCAL 1 / slave2 >> 2015/02/19 11:40:05 26 ms >> 26.0 B >> 3 216 0 SUCCESS PROCESS_LOCAL 0 / slave1 >> 2015/02/19 11:40:05 11 ms >> 0.0 B >> 2 215 0 SUCCESS PROCESS_LOCAL 2 / slave3 >> 2015/02/19 11:40:05 27 ms >> 26.0 B >> 7 220 0 SUCCESS PROCESS_LOCAL 0 / slave1 >> 2015/02/19 11:40:05 11 ms >> 0.0 B >> 10 223 0 SUCCESS PROCESS_LOCAL 2 / slave3 >> 2015/02/19 11:40:05 23 ms >> 26.0 B >> 6 219 0 SUCCESS PROCESS_LOCAL 2 / slave3 >> 2015/02/19 11:40:05 23 ms >> 26.0 B >> 9 222 0 SUCCESS PROCESS_LOCAL 3 / slave4 >> 2015/02/19 11:40:05 23 ms >> 26.0 B >> 8 221 0 SUCCESS PROCESS_LOCAL 1 / slave2 >> 2015/02/19 11:40:05 23 ms >> 26.0 B >> 11 224 0 SUCCESS PROCESS_LOCAL 0 / slave1 >> 2015/02/19 11:40:05 10 ms >> 0.0 B >> 14 227 0 SUCCESS PROCESS_LOCAL 2 / slave3 >> 2015/02/19 11:40:05 24 ms >> 26.0 B >> 13 226 0 SUCCESS PROCESS_LOCAL 3 / slave4 >> 2015/02/19 11:40:05 23 ms >> 26.0 B >> 16 229 0 SUCCESS PROCESS_LOCAL 1 / slave2 >> 2015/02/19 11:40:05 22 ms >> 26.0 B >> 12 225 0 SUCCESS PROCESS_LOCAL 1 / slave2 >> 2015/02/19 11:40:05 22 ms >> 26.0 B >> 15 228 0 SUCCESS PROCESS_LOCAL 0 / slave1 >> 2015/02/19 11:40:05 10 ms >> 0.0 B >> 17 230 0 SUCCESS PROCESS_LOCAL 3 / slave4 >> 2015/02/19 11:40:05 22 ms >> 26.0 B >> 23 236 0 SUCCESS PROCESS_LOCAL 0 / slave1 >> 2015/02/19 11:40:05 10 ms >> 0.0 B >> 22 235 0 SUCCESS PROCESS_LOCAL 2 / slave3 >> 2015/02/19 11:40:05 21 ms >> 26.0 B >> 19 232 0 SUCCESS PROCESS_LOCAL 0 / slave1 >> 2015/02/19 11:40:05 10 ms >> 0.0 B >> 21 234 0 SUCCESS PROCESS_LOCAL 3 / slave4 >> 2015/02/19 11:40:05 25 ms >> 26.0 B >> 18 231 0 SUCCESS PROCESS_LOCAL 2 / slave3 >> 2015/02/19 11:40:05 24 ms >> 26.0 B >> 20 233 0 SUCCESS PROCESS_LOCAL 1 / slave2 >> 2015/02/19 11:40:05 28 ms >> 26.0 B >> 25 238 0 SUCCESS PROCESS_LOCAL 3 / slave4 >> 2015/02/19 11:40:05 20 ms >> 26.0 B >> 28 241 0 SUCCESS PROCESS_LOCAL 1 / slave2 >> 2015/02/19 11:40:05 27 ms >> 26.0 B >> 27 240 0 SUCCESS PROCESS_LOCAL 0 / slave1 >> 2015/02/19 11:40:05 10 ms >> 0.0 B >> >> >> Thanks >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Some-tasks-taking-too-much-time-to-complete-in-a-stage-tp21724.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > -- Regards, Jatinpreet Singh
