OIiveirra opened a new pull request, #69408:
URL: https://github.com/apache/doris/pull/69408
### What problem does this PR solve?
Issue Number: None
Related PR: #66627
Problem Summary:
On branch-4.2, removing a partition field and then dropping its source
column can leave retained Iceberg partition specs referring to a field that no
longer exists in the schema. Other engines can then fail to resolve the
historical partition type and read the table.
Reject top-level and nested column drops when any retained partition spec
references the target source ID or a descendant of the target struct. This
includes historical specs without live files and format-v1 void fields. For
example, after `DROP PARTITION KEY metric`, `DROP COLUMN metric` is rejected
while the retained spec still references it.
Pin the table UUID and target field IDs across refresh, and stage the
standard SDK schema update against the validated metadata base. Direct catalogs
retain metadata CAS; REST commits add the standard last-assigned-partition-ID
requirement to reject concurrent new partition references. The REST
precondition does not modify partition specs, so unrelated columns remain
droppable consecutively. Preserve branch-4.2 session authentication, the
writable catalog lease, writer-property/name-mapping updates, and reserved
row-lineage-name validation against refreshed metadata.
### Release note
Reject Iceberg column drops when the column or a nested field beneath it is
referenced by any retained partition spec, preventing invalid historical
partition metadata.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Validation:
- 353 focused FE unit tests passed with zero failures, errors, or skips,
including real SDK REST HTTP requests, concurrent partition changes,
consecutive drops, delegated authentication, and refreshed row-lineage
validation.
- FE Checkstyle, `git diff --check`, and a standard clang21/ASAN FE+BE
build passed with JDK17.
- `test_iceberg_delete_unpartitioned_evolution` passed both output
generation and normal comparison: 1 suite, 0 failed/fatal/skipped each. It
covers consecutive ordinary drops, historical-spec rejection for Parquet/ORC,
mixed specs, and historical binary/timestamp data. This used isolated REST/S3
with Spark 4.0.0/Iceberg 1.10.1 and Doris Iceberg 1.11.0; the original
HMS/HDFS/Trino environment was not rerun.
- Behavior changed:
- [ ] No.
- [x] Yes. Reject column drops referenced by retained partition specs
before committing the schema update; allow unrelated column drops.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
