GitHub user liutang123 opened a pull request: https://github.com/apache/spark/pull/20184
[SPARK-22987][Core] UnsafeExternalSorter cases OOM when invoking `getIterator` function. ## What changes were proposed in this pull request? ChainedIterator.UnsafeExternalSorter remains a Queue of UnsafeSorterIterator. When call `getIterator` function of UnsafeExternalSorter, UnsafeExternalSorter passes an ArrayList of UnsafeSorterSpillReader to the constructor of UnsafeExternalSorter. But, UnsafeSorterSpillReader maintains a byte array as buffer, witch capacity is more than 1 MB. When spilling frequently, this case maybe causes OOM. I try to change the Queue in ChainedIterator to a Iterator. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liutang123/spark SPARK-22987 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20184 ---- commit d57ce865729ce4d0d84a0fee0edf4dd6febe54bc Author: liutang123 <liutang123@...> Date: 2018-01-08T04:33:51Z [SPARK-22987][Core] UnsafeExternalSorter cases OOM when invoking `getIterator` function. ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org