[ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138354#comment-14138354 ]
Sandy Ryza commented on SPARK-3577: ----------------------------------- In the old code, the ShuffleWriteMetrics didn't get passed into the disk writer. The disk writer would track the bytes it wrote itself and how long it took to write them and these would be used to create the shuffle metrics. ExternalSorter didn't interact with ShuffleWriteMetrics at all. With the change, ExternalSorter creates a ShuffleWriteMetrics because it's now what's used to find how many bytes were written from the disk writer. It might make sense to rename ShuffleWriteMetrics to something like WriteMetrics or DiskWriteMetrics, though I think this was discussed and rejected in the initial patch. > Shuffle write time incorrect for sort-based shuffle > --------------------------------------------------- > > Key: SPARK-3577 > URL: https://issues.apache.org/jira/browse/SPARK-3577 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.1.0 > Reporter: Kay Ousterhout > Priority: Blocker > > After this change > https://github.com/apache/spark/commit/4e982364426c7d65032e8006c63ca4f9a0d40470 > (cc [~sandyr] [~pwendell]) the ExternalSorter passes its own > ShuffleWriteMetrics into ExternalSorter. The write time recorded in those > metrics is never used -- meaning that when someone is using sort-based > shuffle, the shuffle write time will be recorded as 0. > [~sandyr] do you have time to take a look at this? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org