haohuaijin commented on PR #24705: URL: https://github.com/apache/datafusion/pull/24705#issuecomment-5441498249
Results from stacking [apache/arrow-rs#10859](https://github.com/apache/arrow-rs/pull/10859) (same-projection filter fusion) on this branch, benchmarked on [PR #24705](https://github.com/apache/datafusion/pull/24705). **What PR #24705 carries** (on top of #24366's arrow-rs main pin): - arrow/parquet pinned to arrow-rs main + [#10859](https://github.com/apache/arrow-rs/pull/10859): consecutive `RowFilter` predicates with the same single-column projection are fused into one decoded batch stream (the shared column is decoded once, later predicates see only surviving rows) - `pushdown_filters` default flipped to `true` - this branch merged in (`progressive_io` one-shot row-group I/O + the narrow-projection `pushdown_filter_mode` gate) - `106653a` re-applied (`pushdown_dynamic_filters_only`: with a dynamic filter on a narrow projection, the RowFilter is built per-conjunct instead of all-or-nothing) - a per-conjunct refinement on top ([c613c081](https://github.com/haohuaijin/arrow-datafusion/commit/c613c081a56abda91f708ab10771b04244cfe316), details in point 2 below) **1. Fusion fixes Q25.** When consecutive `RowFilter` predicates share the same single-column projection, the reader evaluates them against one decoded batch stream instead of decoding the column once per predicate. Q25 (static `"SearchPhrase" <> ''` + TopK threshold, both on `SearchPhrase`) goes from ~1.27x slower on this branch's head (narrow-projection gate, no fusion — "gated head" below) to **no change** vs pushdown-off. **2. Fusion makes the reverted `106653a` viable.** The [local Q25 experiment](https://github.com/apache/datafusion/pull/24426#issuecomment-5342167688) showed dynamic-only lost because gating the static `<> ''` removes the only effective scan-level pruning — a forced choice only because pushing the same-column static also cost a second decode pass. Fusion removes that cost, enabling a per-conjunct rule ([c613c081](https://github.com/haohuaijin/arrow-datafusion/commit/c613c081a56abda91f708ab10771b04244cfe316)): with a dynamic filter present, a static conjunct stays pushed iff its only column matches the sole column of a dynamic conjunct; the rest are declined. Results (3 `clickbench_partitioned` runs, min-based, vs pushdown-off merge-base): | Query | gated head (this branch, no fusion) | + fusion + per-conjunct gate | |---|---|---| | Q24 | 1.33x slower | **+1.37x faster** | | Q25 | 1.27x slower | **no change** | | Q26 | 1.39x slower | 1.15x slower (two-column sort key → dynamic-only) | | Overall | 4 slower | **1 slower / 6 faster / 36 unchanged, total -15%**, Q23 keeps +20x | the result maybe interesting to you cc @alamb @zhuqi-lucas @hhhizzz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
