GauravWaghmare opened a new issue, #26809:
URL: https://github.com/apache/datafusion/issues/26809
### Is your feature request related to a problem or challenge?
Ungrouped / already push a **dynamic filter** into parquet scans (), but
they do **not** ask the scan to visit files / row groups in best-first order.
does, via → (, , dynamic RG pruning). only runs when a is present, so
never takes that path.
On unsorted parquet (no / Exact output ordering), that visit-order gap
dominates. A real workload (timestamp column, extra equality , Inexact file
stats) measured:
| Query | Scan extras | First files in | Scan rows | wall |
|---|---|---|---|---|
| | , , | last hour of the range, ~8MB files | 110k | **~1.4s** |
| (same ) | only, **no** reorder | earlier hour, ~250–350MB files | 12k |
**~5.0s** |
returned **fewer** rows and was still ~3× slower because it walked listing
order. Dynamic prune cannot help until a large value has been seen.
A residual makes this worse: does not implement (default ), so even TopK
loses reorder if the predicate is not fully absorbed into the scan. Filters are
order-preserving; Projection / CoalesceBatches already delegate.
### Describe the solution you'd like
Keep . Do **not** SQL-rewrite to .
1. **Ungrouped single-column /** (optional , no other aggs that would make a
single order illegal): attach an **Inexact** sort request to the input scan,
using the same pipeline parquet already has.
- → (SQL skips nulls)
- →
2. Combined with the existing aggregate dynamic filter, the scan should see
high- (or low-) row groups first and prune the rest.
3. ** should be transparent** for (delegate to children), so a residual
predicate does not drop reorder for TopK *or* min/max.
(Exact min/max → literal, no scan) should remain preferred when stats are
Exact. This request is for the **Inexact** parquet / Delta path.
### Describe alternatives you've considered
**Rewrite to .** Rejected as the primary approach:
- Empty input: returns one row; returns zero rows.
- Bare is under default ; is skip-nulls / .
- cannot share one TopK.
A rewrite could still be used internally if empty-input and null-ordering
are preserved (e.g. keep on top of a fetch-1 scan). Pushing sort into the scan
under is the smaller, equivalent change.
**Rely only on aggregate dynamic filters.** Already implemented;
insufficient without visit order, as in the table above.
**Document “write instead of .”** Works only when the user can change SQL
and accept empty-set semantics.
### Additional context
Relevant code:
- only walks ()
- Inexact reorder ()
- Ungrouped min/max dynamic filter init ( in )
- has no override (; default in is )
/ mixed aggregates should stay out of scope unless every aggregate is
compatible with one order.
### Willingness to contribute
I would like to contribute to this feature, but request guidance from the
DataFusion community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
