hubgeter opened a new pull request, #70016:
URL: https://github.com/apache/doris/pull/70016
### What problem does this PR solve?
Related PR: #69678
Problem Summary:
An internally partition-pruned Iceberg FileScan still derives its statistics
from the whole table. For example, an MV compensation scan that selects one of
three one-row partitions is estimated at three rows, so its scan CPU cost is 15
rather than 5 with the existing cost factor. Re-pruning preserves the selected
partitions but does not correct this estimate.
Retain snapshot-consistent partition row estimates in the existing connector
MVCC partition view and its converted-view cache, then use the selected
partitions in both logical and physical FileScan statistics. Sum physical
partition rows across partition evolution and keep column statistics valid for
the selected row count, including a known empty selection. Use the scan
reference's pinned snapshot without loading deferred metadata from statistics
calculation. Unknown counts, delete files, native/deferred selections and
unavailable views retain the existing estimation path. Scan cost factors are
unchanged.
The connector carrier gains an optional row estimate, so the connector
plugin API is bumped from 28.0 to 29.0 with its recorded surface updated;
connector plugins must be rebuilt against the matching API. Metadata-listing
close failures propagate instead of caching an incomplete partition view as
successful.
Validation:
- Checkstyle passed for fe-connector-spi, fe-connector-iceberg and fe-core,
including test sources.
- The connector SPI suite passed (155 tests), including API surface gates.
- The final standard `run-fe-ut.sh` run covering the SPI suite, Iceberg
partition metadata and FE statistics/cache/plugin tests is in progress. Results
will be updated here when it finishes.
- External-cluster regression tests and a full FE package build have not
been run for this Apache adaptation.
### Release note
Improve scan row and cost estimates for internally partition-pruned Iceberg
scans, including materialized view compensation. Connector plugins require API
version 29.0.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Eligible scans use selected-partition row estimates; final
plan selection remains cost based.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
