thank you very much, Andrew by the difinition of "Fetch Wait Time", can i make a conclusion that task pipelines block fetch and job doing?
Andrew Or-2 wrote > Hi wxhsdp, > > These times are computed from Java's System.currentTimeMillis(), which is > "the > difference, measured in milliseconds, between the current time and > midnight, January 1, 1970 UTC." Thus, this quantity doesn't mean much by > itself, but is only meaningful when you subtract it from another > System.currentTimeMillis() to find the time elapsed. For instance, in your > case (Finish Time - Launch Time) = 1862, which means it took 1862 ms for > the task to complete (but the actual execution only took 1781 ms, the rest > being overhead). > > Correct, (Shuffle Finish Time - Launch Time) is the total time it took for > this task to fetch blocks, and (Finish Time - Shuffle Finish Time) is the > actual execution after fetching all blocks. > > "Fetch Wait Time" on the other hand is the time spent blocking on the > thread to wait for shuffle blocks while not doing anything else. For > instance, the example given in the code comments is: "if block B is being > fetched while the task is not finished with processing block A, it is not > considered to be blocking on block B." > > "Shuffle Write Time" is the time written to write the shuffle files (only > for ShuffleMapTask). It is in nanoseconds, which is slightly inconsistent > with other values in these metrics. > > By the way, all of this information is available in the code comments: > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala > > > > > On Tue, May 6, 2014 at 11:10 PM, wxhsdp < > wxhsdp@ > > wrote: > >> any ideas? thanks! >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/details-about-event-log-tp5411p5476.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/details-about-event-log-tp5411p5624.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
