Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19184 @jerryshao Actually the second half of your comment is not valid in this case. The PR is not targeting the merge sort in this case, but relevant when iterating over all tuples. `UnsafeExternalSorter` has two methods to iterate over the tuples. You are referring to `getSortedIterator` - which uses a PriorityQueue and requires all files to be opened at the same time (so that it can return a sorted iterator). The primary usecase of this PR is for `getIterator` - where we are simply iterating over all tuples : and used in `ExternalAppendOnlyUnsafeRowArray` for example : there is no need to sort here. This is used by various `WindowFunctionFrame` implementations for example.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org