GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/18679
[SPARK-21319][SQL] Fix memory leak in sorter
## What changes were proposed in this pull request?
`UnsafeExternalSorter.recordComparator` can be either `KVComparator` or
`RowComparator`, and both of them will keep the reference to the input rows
they compared last time.
After sorting, we return the sorted iterator to upstream operators.
However, the upstream operators may take a while to consume up the sorted
iterator, and `UnsafeExternalSorter` is registered to `TaskContext` at
[here](https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L159-L161),
which means we will keep the `UnsafeExternalSorter` instance and keep the last
compared input rows in memory until the sorted iterator is consumed up.
Things get worse if we sort within partitions of a dataset and coalesce all
partitions into one, as we will keep a lot of input rows in memory and the time
to consume up all the sorted iterators is long.
This PR propose to remove the reference of the input rows in the comparator
after the sort is finished.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark memory-leak
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18679.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18679
----
commit f5752d4a1ed5b6dc1b1d2dea4056ff8ebb65f657
Author: Wenchen Fan <wenc...@databricks.com>
Date: 2017-07-19T08:13:48Z
Fix memory leak in sorter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
