spark git commit: [SPARK-16827] Stop reporting spill metrics as shuffle metrics

[andrewor14]

Repository: spark
Updated Branches:
  refs/heads/master 2b01d3c70 -> e56614cba


[SPARK-16827] Stop reporting spill metrics as shuffle metrics

## What changes were proposed in this pull request?

Fix a bug where spill metrics were being reported as shuffle metrics. 
Eventually these spill metrics should be reported (SPARK-3577), but separate 
from shuffle metrics. The fix itself basically reverts the line to what it was 
in 1.6.

## How was this patch tested?

Tested on a job that was reporting shuffle writes even for the final stage, 
when no shuffle writes should take place. After the change the job no longer 
shows these writes.

Before:
![screen shot 2016-10-03 at 6 39 59 
pm](https://cloud.githubusercontent.com/assets/1514239/19085897/dbf59a92-8a20-11e6-9f68-a978860c0d74.png)

After:
<img width="1052" alt="screen shot 2016-10-03 at 11 44 44 pm" 
src="https://cloud.githubusercontent.com/assets/1514239/19085903/e173a860-8a20-11e6-85e3-d47f9835f494.png";>

Author: Brian Cho <b...@fb.com>

Closes #15347 from dafrista/shuffle-metrics.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e56614cb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e56614cb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e56614cb

Branch: refs/heads/master
Commit: e56614cba99bfdf5fa8a6c617fdd56eca2b34694
Parents: 2b01d3c
Author: Brian Cho <b...@fb.com>
Authored: Fri Oct 7 11:37:18 2016 -0400
Committer: Andrew Or <andrewo...@gmail.com>
Committed: Fri Oct 7 11:37:18 2016 -0400

----------------------------------------------------------------------
 .../spark/util/collection/unsafe/sort/UnsafeExternalSorter.java  | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/e56614cb/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
----------------------------------------------------------------------
diff --git 
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
 
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
index 428ff72..7835017 100644
--- 
a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
+++ 
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
@@ -145,7 +145,9 @@ public final class UnsafeExternalSorter extends 
MemoryConsumer {
     // Use getSizeAsKb (not bytes) to maintain backwards compatibility for 
units
     // this.fileBufferSizeBytes = (int) 
conf.getSizeAsKb("spark.shuffle.file.buffer", "32k") * 1024;
     this.fileBufferSizeBytes = 32 * 1024;
-    this.writeMetrics = taskContext.taskMetrics().shuffleWriteMetrics();
+    // The spill metrics are stored in a new ShuffleWriteMetrics, and then 
discarded (this fixes SPARK-16827).
+    // TODO: Instead, separate spill metrics should be stored and reported 
(tracked in SPARK-3577).
+    this.writeMetrics = new ShuffleWriteMetrics();
 
     if (existingInMemorySorter == null) {
       this.inMemSorter = new UnsafeInMemorySorter(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org