yashrb24 commented on PR #25358: URL: https://github.com/apache/datafusion/pull/25358#issuecomment-6415398490
@kosiew thanks for the feedback and sorry for the delayed response! So a bunch of observations from some of my attempts to do automatic staging: 1. A lot of the numbers depend on what one categorizes as a "cheap key" and how we define a stage. We do have some [precedence](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/simplify_expressions/reorder_predicates.rs) for defining this set of cheap keys, but it exists in the logical planning step and not sure if we want to reproduce all of that infra here too. If this is the direction you are aligned on, I'll work on this. 2. One of the drawbacks with this approach of "split by key cost" that I can see is that it will not help in cases where we have multiple expensive order-by expressions stacked together (eg: `order by expr(col1), expr(col2) ... expr(coln)` where expr is some expensive operation). This was originally the reason of going with the fixed grouped size. Again a lot of this will depend on the distribution of data as you have highlighted. 3. The "Adapt at runtime" is a useful fallback, I'll have this implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
