xylaaaaa opened a new pull request, #70173:
URL: https://github.com/apache/doris/pull/70173
### What problem does this PR solve?
Issue Number: None
Related PR: #69392
Problem Summary:
Backport #69392 to branch-4.2. Iceberg DELETE/UPDATE/MERGE could record
position-delete and Puffin files under BE's normalized `s3://` address even
when the table uses `oss://`, `s3a://`, or `abfss://`. These metadata paths can
prevent Doris and other engines from reading the files.
BE now derives physical I/O paths and raw metadata paths from one output
location for data files, position deletes, and deletion vectors. FE registers
the raw locations from the actual bound sinks and validates the entire
collected file list before staging INSERT/overwrite, DELETE, MERGE, or rewrite
updates.
Release-branch adaptation:
- Use the existing typed Iceberg transaction and planner sinks. Validate at
transaction completion rather than in the report RPC, since these branches do
not have #66348's report ACK mechanism.
- Register rewrite group locations after binding the shared transaction;
preserve empty static partition overwrites, which execute without a BE sink.
- Preserve the branch's existing failed-file cleanup behavior. This backport
does not add ACK-based ownership transfer or rollback file deletion.
- No wire-format changes; the Thrift changes are comments only.
- Adapt the branch-specific unsupported-Parquet-codec test to the new
output-location object.
### Release note
Fix Iceberg files being committed with normalized storage addresses instead
of their raw table data locations. Reject out-of-location file paths before
staging Iceberg updates. Upgrade BE before FE: an older BE that reports
normalized delete-file paths will be rejected by the upgraded FE. Failed writes
can still leave uncommitted files under the release branch's existing cleanup
behavior.
### Check List (For Author)
- Test:
- [x] Unit Test: 83 focused FE tests passed via `run-fe-ut.sh --run`,
covering transactions, all four completion paths, raw/normalized and
directory-prefix rejection, sink registration, empty static overwrite, rewrite
groups, and SQL failure propagation.
- [x] Java Checkstyle passed for the final changed files; C++
clang-format 16 checks passed.
- [ ] BE unit tests: attempted ASAN build, but local macOS/Clang
compatibility errors prevented execution. CI validation is still needed.
- [ ] Iceberg regression: cases are included; not run locally because
the external fixture/Docker daemon is unavailable.
- C++ static analysis was attempted; the full run is not green due to
existing macro/NOLINT/complexity diagnostics. No broad suppressions or
environment changes are included.
- Behavior changed:
- [x] Yes: metadata paths use raw locations, and invalid reported paths
fail the statement.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
