athlcode opened a new pull request, #25328:
URL: https://github.com/apache/datafusion/pull/25328
## Which issue does this PR close?
- Closes #24944.
## Rationale for this change
Parquet row group, page index and bloom filter pruning only happens when a
scan executes, so the optimizer plans with statistics for entire files. When
data is clustered on a filtered column, the estimates can be far off. On the
IMDB `movie_info` table sorted by `info`, `info IN ('Bulgaria')` is estimated
at 14.8M rows, while page index pruning narrows the scan to 15,360 rows. Better
estimates at planning time can improve join selection and enable follow-ups
such as eager fetching of small scans (#24922).
## What changes are included in this PR?
- New options `datafusion.execution.parquet.eager_pruning` (`disabled`
(default), `row_groups`, `page_index`, `bloom_filters`) and
`eager_pruning_file_limit` (default 256). Both can also be set per table via
`format.` options.
- **API change:** `FileFormat::create_physical_plan` takes a new `filters:
&[Arc<dyn PhysicalExpr>]` argument. `ListingTable` passes its non-partition
filters as hints.
- When enabled, `ParquetFormat` runs the same pruning stages during planning:
- drops files where no row can match,
- attaches the resulting `ParquetAccessPlan` to each file, with fully
matched flags cleared since the scan may execute with a different predicate,
- rebuilds file statistics as inexact from the row groups that are read.
- `EXPLAIN` shows the outcome, e.g. `eager_pruning=[level=row_groups,
files_pruned=0/1, row_groups_pruned=8/10, rows_pruned=800/1000]`.
- Refactor: the bloom filter loading and file schema coercion are shared
between the opener and planning.
- Proto fields for the new options, plus updates to `configs.md` and the
56.0.0 upgrade guide.
## What is the testing strategy for this PR?
- Unit tests in `datasource-parquet/src/eager_pruning.rs` (statistics rules,
file limit, fully matched flags, display).
- Integration tests in `core/tests/parquet/eager_pruning.rs`:
- per-level estimates, file pruning, file limit, per-table options,
partition columns,
- results identical to pruning disabled at every level, with filter
pushdown on and off,
- correct results when attached access plans are lost (as with plan
serialization),
- a page index pass that prunes nothing leaves files unchanged,
- `collect_statistics = false`.
- `parquet_eager_pruning.slt` covering EXPLAIN output for each level.
- Proto round-trip tests, including messages without the new fields.
- IMDB benchmark (local disk): on sorted `movie_info`, the estimate goes
from 14.8M to 15,360 rows at `page_index`, with under 0.5 ms of added planning.
On this data query 3b got about 10% slower because the better estimates changed
the join plan. That is worth discussing.
## Are there any user-facing changes?
- New config options. Eager pruning is disabled by default, so nothing
changes unless it is enabled.
- **Breaking API change** to `FileFormat::create_physical_plan` (needs the
`api change` label). Migration notes are in the upgrade guide.
- Known limitations: encrypted files and tables with virtual columns are
skipped. Execution still re-evaluates page indexes and bloom filters; a
follow-up issue will cover avoiding that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
