Hey Umar, I dug into this a bit today out of curiosity since I also wasn't sure.
I updated the in-line documentation here: https://github.com/apache/incubator-spark/pull/209/files The more important metric is `fetchWaitTime` which indicates how much of the task runtime was spent waiting for input data. remoteFetchTime is an aggregation of all of the fetch delays for each block... this second metric is a bit more convoluted because those fetches can actually overlap, so if this is high it doesn't necessarily indicate any latency hit. - Patrick On Mon, Nov 25, 2013 at 1:23 PM, Umar Javed <umarj.ja...@gmail.com> wrote: > Any clarification on this? thanks. > > > On Wed, Nov 20, 2013 at 3:02 PM, Umar Javed <umarj.ja...@gmail.com> wrote: > >> In the class ShuffleReadMetrics in executor/TaskMetrics.scala, there are >> two variables: >> >> 1) fetchWaitTime: /** >> >> >> * Total time that is spent blocked waiting for shuffle to fetch data >> >> >> */ >> >> 2) remoteFetchTime >> >> /** >> >> >> * The total amount of time for all the shuffle fetches. This adds up >> time from overlapping >> >> * shuffles, so can be longer than task time >> >> >> */ >> >> As I understand it, the difference between these two is that fetchWaitTime >> is remoteFetchTime without the overlapped time counted exactly once. Is >> that right? Can somebody explain the difference better? >> >> thanks! >>
