peter-toth commented on PR #14948: URL: https://github.com/apache/iceberg/pull/14948#issuecomment-4164530747
> > As we discussed offline, a long term (after [apache/spark#54330](https://github.com/apache/spark/pull/54330)) solution could be to improve the new GroupPartitionsExec operator to not only coalesce partitions with the same key, but k-way merge them to keep their sorted order. > > Thanks @peter-toth , this makes sense. I think this PR is still needed and would still be valuable for tables with decently sized partitions. Since it's gated by a flag, I think it's safe to implement. Absolutely. FYI https://github.com/apache/spark/pull/54330 has been merged. https://github.com/apache/spark/pull/55116 will do the Spark side k-way merge to keep full ordering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
