mhamedbenjmaa opened a new issue, #7877:
URL: https://github.com/apache/hop/issues/7877
### What would you like to happen?
## Problem
Two bottlenecks limit `SortRows` throughput on large datasets:
1. **In-memory chunk sort is single-threaded.** `quickSort()` calls
`List.sort()` (TimSort) on the buffer — runs on one core regardless of
available CPUs, even for large in-memory buffers.
2. **Disk merge is O(k) per row instead of O(log k).** When rows spill to
disk, the final k-way merge in `getBuffer()` keeps candidate rows
(`SortRowsData.tempRows`) in a plain `ArrayList<RowTempFile>`, using
`Collections.binarySearch` + `ArrayList.add(index, …)` /
`removeFirst()` to insert/extract the next row
Together these mean neither phase of the sort scales with dataset size or
available hardware the way it should.
## Proposal
1. **Parallelize the in-memory sort**: use
`Arrays.parallelSort(Object[][], comparator)` instead of `List.sort()`
for the buffer, at least above some size threshold. Stable, same
output ordering as today — just uses multiple cores.
2. **Replace the merge structure with a heap**: swap `data.tempRows` for
a `java.util.PriorityQueue<RowTempFile>` in the merge loop. Drop-in,
no behavior change.
## Scope
- `SortRows.java`: `quickSort()` (parallel sort), `getBuffer()` (heap merge)
- `SortRowsData.java`: `tempRows` field type
- Tests: correctness with large buffers (parallel sort) and many spill
files (heap merge); confirm output ordering unchanged in both cases
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
