zhuqi-lucas opened a new issue, #24270: URL: https://github.com/apache/datafusion/issues/24270
Follow-up to #23036, which tracked the broader **Sort Pushdown** effort across DataFusion v52 → v55. The bulk of that epic is now complete — the optimizer-rule framework and `Exact / Inexact / Unsupported` classifier, reverse iteration, statistics-based file/RG reorder, multi-partition `BufferExec`, and the runtime RG-level early stop (#22450, merged) have all landed. This issue tracks the remaining open follow-ups so #23036 can be closed as substantially done. See #23036 for the full architecture, phase history, and benchmark results. ## Remaining open items ### Statistics-based reorder follow-ups - [ ] #22198 — Follow-ups for stats-based RG / file reorder: multi-column, function-wrapped sort expressions, compound `ORDER BY` - [ ] #21169 — Bug: `EnforceDistribution` loses `fetch` (LIMIT) from `CoalescePartitionsExec` / `SortPreservingMergeExec` - [ ] #19394 — Add more `ExecutionPlan` nodes to support sort pushdown ### Multi-partition / BufferExec follow-ups - [ ] #21381 — Parallel merge in `SortPreservingMergeExec` after sort elimination - [ ] #21440 — Dynamic `BufferExec` sizing: row limit + memory cap - [ ] #21780 — Refactor: make `DynamicFilterPhysicalExpr.fetch` a shared reference to `SortExec.fetch` ### Runtime RG-level early stop follow-ups (Phase 4 landed via #22450) - [ ] #23067 — Per-RG `fully_matched` `RowFilter` skip on top of #22450 (PR #23696, in review) - [ ] #22949 — Bug: `PhysicalExprSimplifier` folds `DynamicFilterPhysicalExpr` wrapper, breaking the live link to TopK updates ### Future — `Exact` reverse path (blocked on arrow-rs page-level reverse) - [ ] #19486 — Support reverse page for sort pushdown (phase 1 follow-up) - [ ] #19048 — Reverse parquet OOM mitigation for huge row groups - [ ] Cross-repo: [apache/arrow-rs#9937](https://github.com/apache/arrow-rs/issues/9937) — POC `ReverseSerializedPageReader` (unlocks strictly-DESC output → `Exact` sort pushdown) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
