zanmato1984 commented on issue #47832: URL: https://github.com/apache/arrow/issues/47832#issuecomment-6452732093
Thanks for the ping, @AlenkaF. I would lean toward keeping `sort_indices` as a synchronous compute primitive and letting Acero handle parallel scheduling and concurrency control, rather than having the compute function manage its own parallelism. `Table.sort_by` could then delegate to an Acero plan, similar to what `Table.group_by(...).aggregate(...)` already does. This would require adding parallel sorting to Acero's `order_by` first. Although Acero provides a framework for parallel execution, implementing a parallel sort is non-trivial because sorting is not batch-independent. Locally sorted batches still need to be merged into a globally ordered result. Parallel run generation followed by a merge tree seems like a straightforward starting point, although the final merge could remain a bottleneck. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
