alamb commented on PR #24426: URL: https://github.com/apache/datafusion/pull/24426#issuecomment-5333084040
## Status update This PR now carries three logical changes (see description): `progressive_io` one-shot row-group I/O (#24395), the narrow-projection pushdown gate ported from #23420 (`pushdown_filter_mode`, thanks @zhuqi-lucas), and a refinement (`106653a`) that gates only *static* conjuncts when a dynamic (TopK/aggregate) filter is present, so the dynamic threshold still reaches the scan while e.g. `SearchPhrase <> ''` stays in the `FilterExec` above. ### What the benchmark runs so far show (comparing repeated runs to separate signal from noise) - The gate eliminates the broad pushdown regressions (Q7/Q10/Q11/Q14/Q17/Q21/Q22/Q34 in earlier runs), and Q23 keeps its ~20x (plain) / ~6x (simulated latency) improvement throughout. - With `106653a` (dynamic-only gating): **Q24 flips to +1.43x faster** and Q26 drops into the noise band — the TopK threshold on `EventTime` (a cheap int64 compare) prunes rows before decoding `SearchPhrase`. - **Q25 got worse** (~1.45x slower): it sorts on `SearchPhrase` itself, so the dynamic threshold is a per-row UTF-8 comparison on a column that must be decoded anyway — no decode savings, pure compare overhead. Theory: a dynamic threshold is only worth pushing as a *row* filter when its column is cheap to compare and/or pruning it avoids decoding other columns; string-keyed thresholds on narrow projections should rely on row-group-level pruning only. - **Q41/Q42 under simulated latency** show a stubbornly consistent ~1.25x (~+28ms) across all three latency runs, unaffected by either gate variant. The near-constant absolute delta looks like one extra round trip somewhere in the pushdown path (aggregate-TopK dynamic filter is a suspect). Not yet explained. ### Next steps Building `--profile=profiling` binaries at the merge-base, `026fb7d`, and `106653a` to reproduce these locally and profile: `EXPLAIN ANALYZE` on q24/q25/q26/q41/q42, object-store request counts, and one-variable-at-a-time toggles of `pushdown_filter_mode`, `progressive_io`, and `enable_aggregate_dynamic_filter_pushdown`. Results to follow here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
