Off the top of my head, the most likely cause would be driver GC issues. You can diagnose this by enabling GC printing at the driver and you can fix this by increasing the amount of memory your driver program has (see http://spark.apache.org/docs/0.9.0/tuning.html#garbage-collection-tuning).
The "launch time" statistic would also be useful -- if all tasks are launched at around the same time and complete within 300ms, yet the total time is 10s, that strongly suggests that the overhead is coming before the first task is launched. Similarly, it would be useful to know if there was a large gap between launch times, or if it appeared that the launch times were serial with respect to the durations. If for some reason Spark started using only one executor, say, each task would take the same duration but would be executed one after another. On Tue, Apr 8, 2014 at 8:11 AM, Yana Kadiyska <yana.kadiy...@gmail.com>wrote: > Hi Spark users, I'm very much hoping someone can help me out. > > I have a strict performance requirement on a particular query. One of > the stages shows great variance in duration -- from 300ms to 10sec. > > The stage is mapPartitionsWithIndex at Operator.scala:210 (running Spark > 0.8) > > I have run the job quite a few times -- the details within the stage > do not account for the overall duration shown for the stage. What > could be taking up time that's not showing within the stage breakdown > UI? Im thinking that reading the data in is reflected in the Duration > column before, so caching should not be a reason(I'm not caching > explicitly)? > > The details within the stage always show roughly the following (both > for the 10second and 600ms query -- very little variation, nothing > over 500ms, ShuffleWrite size is pretty comparable): > > StatusLocality LevelExecutor Launch Time DurationGC TimeShuffle Write > 1864 SUCCESSNODE_LOCAL ####### 301 ms 8 ms 111.0 B > 1863 SUCCESSNODE_LOCAL ####### 273 ms 102.0 B > 1862 SUCCESSNODE_LOCAL ####### 245 ms 111.0 B > 1861 SUCCESSNODE_LOCAL ####### 326 ms 4 ms 102.0 B > 1860 SUCCESSNODE_LOCAL ####### 217 ms 6 ms 102.0 B > 1859 SUCCESSNODE_LOCAL ####### 277 ms 111.0 B > 1858 SUCCESSNODE_LOCAL ####### 262 ms 108.0 B > 1857 SUCCESSNODE_LOCAL ####### 217 ms 14 ms 112.0 B > 1856 SUCCESSNODE_LOCAL ####### 208 ms 109.0 B > 1855 SUCCESSNODE_LOCAL ####### 242 ms 74.0 B > 1854 SUCCESSNODE_LOCAL ####### 218 ms 3 ms 58.0 B > 1853 SUCCESSNODE_LOCAL ####### 254 ms 12 ms 102.0 B > 1852 SUCCESSNODE_LOCAL ####### 274 ms 8 ms 77.0 B >
