zhuqi-lucas commented on PR #23696:
URL: https://github.com/apache/datafusion/pull/23696#issuecomment-5379689200
Update on the split:
- **PR 1 — #24509 (strip empty row groups)**: merged. Thanks for the review
there.
- **PR 3 — #24572 (`InitialDecoderState`)**: up and ready, closes #24286.
Pure refactor, landed before the feature so #23696 adds fields to a named
struct instead of reshaping a tuple.
On **PR 2 (prebuild once per file)** — I don't think it stands alone, and
I'd suggest folding the prebuild into PR 4 (#23696) instead. The perf premise
doesn't hold on current `main`: `ParquetPushDecoder::into_builder()` (parquet
59.2.0) carries the `RowFilter` across the rebuild — `builder_from_remaining`
preserves `filter` — so the opener builds the filter once and it survives every
runtime-prune `into_builder().with_row_groups().build()`. There is no
per-row-group filter rebuild on `main` today, so
`prebuild_row_filter_candidates` / `row_filter_from_prebuilt` would have no
consumer and save nothing.
The only per-RG rebuild is #23696's own `fully_matched → not-fully-matched`
toggle. So prebuilding-once only pays off *with* that toggle, and it gives PR 4
a clean, attributable perf story (prebuild + toggle together) rather than
landing unused machinery in a standalone PR 2. I'll still reimplement
`build_row_filter` on top of the two functions there so there's a single
conjunct-split/order/metric path (the drift concern you raised), and pick up
the lazy-metric registration + the `test_input_file_name_projection` revert +
the single `Vec<{index, fully_matched}>` in the same PR.
Does folding PR 2 into PR 4 sound right to you? If so I'll rebase #23696
onto `main` + #24572 once #24572 lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
