NGA-TRAN commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5471579106
Thanks @suremarc for the quick analysis, tests, and suggestions — really appreciate the effort and the momentum it gives this feature request. > If you GROUP BY time_partition, key, date_bin(time) then the aggregate becomes fully streaming. We considered this approach and would use it if necessary, but it’s still sub‑optimal compared with the proposal in this ticket: 1. It requires adding a property that data is sorted on `partition_key, key, timestamp` instead of key, timestamp. 2. Users naturally write `GROUP BY key, date_bin(time)`, and we would need to rewrite it to include `partition_key`. 3. Grouping on three keys is more expensive than grouping on two. 4. Any operators before the group‑by may break the required sort order, leaving us with the cost of a three‑key group‑by without the benefit. The proposed feature feels much more natural: 1. It avoids all the drawbacks above. 2. It fits cleanly into the optimizer: treating non‑overlapping data on group‑by keys as a first‑class property can help not only this case but future optimizations as well. @alamb — this might even be a small research‑driven insight applied to a real‑world workload. It’s not trivial, but meaningful optimizations rarely are. @xavlee, @gene-bordegaray, @jayshrivastava and I have aligned on a reasonable design. Xavier already has four draft PRs: one for tests demonstrating the case, two for plumbing, and one integrating everything. Gene, Jayant, and I will review carefully so Xavier can address all comments before we bring it to the committers. We hope this feature helps push DataFusion to the next level — showcasing a strong query engine ready for critical workloads across the world. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
