asolimando opened a new pull request, #25969:
URL: https://github.com/apache/datafusion/pull/25969
## Which issue does this PR close?
- Closes #25571.
## Rationale for this change
`StatisticsRegistry::default_with_builtin_providers()` registers providers
that are not the default estimation, and most of them reimplement what the
operators already do in `statistics_from_inputs`. They can make estimates worse
(TPC-H Q14: 14.73 billion rows instead of 73,650, see #25570), and fixes in the
operators do not reach their users.
The providers are kept as a place for techniques not yet ready to move into
the operators. Each one starts from the operator's estimate and replaces only
the values its technique estimates.
Integrating the two remaining providers (Filter and Join) into the
operators, as suggested in the issue, is delegated to follow-up PRs. Both
depend on distinct counts, which the Parquet files used by our stats benchmarks
do not carry, so for now they do not change any estimate there and we cannot
evaluate them. We will revisit once NDV values are available in the stats
benchmarks.
## What changes are included in this PR?
- New `StatisticsRegistry::with_experimental_providers()` (Filter and Join).
`default_with_builtin_providers()` is deprecated.
- The Projection, Passthrough, Aggregate, Limit and Union providers are
deprecated, as they duplicate the operators.
- Bug fixes:
- `JoinStatisticsProvider` replaced the operator's estimate with the
Cartesian product when a key had no distinct count (the Q14 estimate above). It
now changes only multi-key inner joins, using the product of the per-key
distinct counts, and delegates otherwise.
- `FilterStatisticsProvider` reduced distinct counts twice. It now applies
the survival model to the input distinct count, capped at the operator's.
- `statistics_registry.slt` uses a test-only provider; its expected output
is unchanged.
## What is the testing strategy for this PR?
- Unit tests for both providers in `operator_statistics/mod.rs`.
- `statistics_registry.slt` passes unchanged; the `join_reorder` example
still flips the build side.
## Are there any user-facing changes?
Deprecations only, see the 56.0.0 upgrade guide. Users of the Filter and
Join providers get the bug fixes.
----
Disclaimer: I used AI to assist in the code generation, I have manually
reviewed the output and it matches my intention and understanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
