Gabriel39 opened a new pull request, #69722:
URL: https://github.com/apache/doris/pull/69722
### What problem does this PR solve?
Backport the requested external-data and FileScannerV2 changes to
`branch-4.2` in their actual master merge order:
| Order | Master PR | Change | Backport |
| --- | --- | --- | --- |
| 1 | #69545 | Preserve Paimon schema options after field renames | Included
|
| 2 | #69551 | Expand Iceberg capability, transaction and snapshot tests |
Included with 4.2 API adaptations |
| 3 | #69614 | Elasticsearch HTTP scans in FileScannerV2 | Already present
through #69644 (`2104b7509cda`); not applied twice |
| 4 | #69612 | Check catalog visibility before SHOW DATABASES accesses
metadata | Included |
| 5 | #69615 | Publish Iceberg Parquet split offsets | Included |
| 6 | #69650 | Preserve ES compound residuals and expand scanner coverage |
Included |
| 7 | #69651 | Accept Elasticsearch 9.x during node discovery | Included |
| 8 | #69646 | Prioritize Parquet V2 row groups using TopN bounds | Included
|
| 9 | #69668 | Arrow loads and schema inference in FileScannerV2 | Included |
The eight outstanding changes are combined in one commit. This branch
retains the 4.2 FE datasource APIs instead of importing master's
connector-module refactor.
### Backport adaptations
- Port Paimon schema restoration, catalog-option provenance, fallback
handling and tests to the existing table-copy/serialization lifecycle. Keep
statement-bound fields when reader options are applied after a schema-only
rename.
- Exercise Iceberg's existing metadata and transaction entry points with
real in-memory tables and retained-generation fixtures. Preserve the nine
disabled known-bug reproductions; omit two connector-only lifecycle tests.
Expiration checks verify live data-file sets and exact delete-file counters;
4.2's separate manifest-name/data-counter issue remains unchanged. The
delete/history SQL suite checks visible rows and eager/late-materialization
parity because 4.2 does not expose master's `_file`/`_pos` columns. The
inventory labels upstream-only evidence explicitly.
- Port ES residual tracking to the legacy planner and add compound OR/NOT,
docvalue eligibility and Thrift-context tests there. Use Java 8-compatible HTTP
APIs for opt-in ES 9 tests and the branch's supported node-address fixtures.
- Keep the existing full-schema Iceberg metadata-table scan, which already
supplies the file-size dependency for split offsets. Adapt the external write
suite to verify offsets without requiring master-only distribution plans or
golden output.
- Use the existing Parquet statistics interface and physical row-location
representation. Keep the TopN scheduling and pruning invariants.
- Include the shared Arrow array normalizer required by the new reader,
taken from the same master revision. Preserve 4.2 Lance/Avro dispatch; omit the
schema-inference test for the unavailable Hive OpenCSV parser.
### Validation
- FE production and test sources compile with the branch's Java 8 target.
Across the targeted runs and focused reruns, 20 test classes report 181
discovered, 171 passed, zero failures/errors, and 10 skipped: nine
intentionally disabled known-bug reproductions plus the opt-in live ES 9 class.
- FE Checkstyle: passed with zero violations.
- Clang compiler syntax checks: all 23 affected C++ translation units passed
against the current sources and freshly generated Thrift, using cached
third-party/generated build dependencies. This is not a full BE link or runtime
test.
- clang-format 16: all 30 affected C++ source/header files passed.
- Groovy 4.0.19 syntax and AST conversion: all seven changed suites passed.
All 105 local inventory links resolve.
- Complete staged-diff self-review, whitespace/conflict checks, and
sensitive-information checks passed.
Full BE linking/runtime tests and SQL/external-service regressions were not
run locally. Live ES 9 tests require their opt-in endpoint. CI is requested for
the combined backport.
### Release note
Fix Paimon reads after field renames, external-catalog SHOW DATABASES
authorization, and ES compound residual filtering. Support ES 9.x node
discovery, publish Iceberg Parquet split offsets, improve Parquet V2 TopN
row-group scheduling, and use FileScannerV2 for Arrow loads and text/JSON
schema inference.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Regression test additions and syntax checks; SQL execution pending
CI
- Behavior changed:
- [x] Yes. Backport the changes listed above.
- Does this need documentation?
- [x] No. Existing interfaces and settings are retained; the test
coverage inventory is included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
