[GitHub] spark pull request #23083: [SPARK-26114][CORE] ExternalSorter Leak

[szhem]

Github user szhem commented on a diff in the pull request:

    https://github.com/apache/spark/pull/23083#discussion_r235319165
  
    --- Diff: 
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala 
---
    @@ -72,7 +73,8 @@ final class ShuffleBlockFetcherIterator(
         maxBlocksInFlightPerAddress: Int,
         maxReqSizeShuffleToMem: Long,
         detectCorrupt: Boolean)
    -  extends Iterator[(BlockId, InputStream)] with DownloadFileManager with 
Logging {
    +  extends Iterator[(BlockId, InputStream)] with DownloadFileManager with 
TaskCompletionListener
    --- End diff --
    
    The main reason is that TaskCompletionListener is added in one place (in 
`initialize` method) and needs to be removed in another one (in `cleanup` 
method).
    
![image](https://user-images.githubusercontent.com/1523889/48833554-abe64780-ed8c-11e8-9da0-ef826918a275.png)
    Will introduce a field for TaskCompletionListener instead. Thank you!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org