Doris-Breakwater commented on issue #69544: URL: https://github.com/apache/doris/issues/69544#issuecomment-6364716066
Read-only triage: the reported interleaving is supported by the Iceberg 1.11.0 commit mechanisms. This is a credible gap in the intended Doris safeguard against dropping retained partition-source fields. It is **not yet an end-to-end Doris REST reproduction or a demonstrated downstream read failure**. Suggested routing: FE / Iceberg REST catalog correctness. The issue currently has no labels. I checked the local `branch-4.2` source at `807c3a02a37d667cc632b6d5096ce2e1e22b8b3b`, the DROP implementation at PR #69408's base `8e590aeb0fe5d6a4a4e11aa3275c2428d7694075`, its reviewed head `9359ffdf0661c627ae00736ade83a8d3c0234ccc`, and the official Iceberg 1.11.0 source artifacts. No repository code was changed and no build or runtime reproduction was run. **Source-supported findings:** 1. The branch/base DROP path calls `updateSchema().deleteColumn(...).commit()` under the writable lease's authenticator; both top-level and nested drops use the ordinary SDK path. Iceberg's `RESTTableOperations.commit()` derives requirements from metadata changes and sends an update request. For a schema DROP, the standard requirements protect UUID, last assigned column ID and current schema ID; they do not compare the complete retained partition-spec set. Adding a last-partition-ID requirement does not turn this into a metadata-generation check. See [Doris DROP](https://github.com/apache/doris/blob/8e590aeb0fe5d6a4a4e11aa3275c2428d7694075/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java#L951-L979) and [SDK requirement construction](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/core/src/main/java/org/apache/iceberg/UpdateRequirements.java#L50-L194). 2. `BaseUpdatePartitionSpec.recycleOrCreatePartitionField()` searches the writer's captured historical specs and can return an existing field without incrementing its counter. `addNonDefaultSpec()` leaves the default spec unchanged; metadata insertion keeps the maximum partition field counter. Thus restoring the prepared `{y,x}` spec can leave UUID, schema, column counter, default spec and partition counter unchanged. The [SDK recycling path](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java#L113-L155) supports the proposed race. The [format specification](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/format/spec.md#partition-evolution) describes 1000 as the v1 reference implementation convention, not a v2 minimum. A counter below 1000 cannot establish absence of partition history. 3. The reference REST handler validates requirements against server metadata, applies the updates, then commits. Its underlying CAS protects that server-side base; it does **not** establish that this is the same generation Doris previously inspected. A restored spec published before the handler reads its base can therefore survive all these checks. On schema changes, `TableMetadata` rebuilds retained specs without validation, while `build()` checks partition compatibility only for the default spec. See [REST commit handling](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L613-L659) and [metadata construction](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1534-L1554). Actual catalog servers may add validation, so their behavior needs testing. 4. A missing current-schema source in a historical spec alone does not prove a crash or an Iceberg format violation. The SDK deliberately preserves historical references, and [`PartitionSpec.partitionType()`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L125-L150) accommodates missing source types using `unknown`. This is consistent with the reported probe not throwing. Any downstream-engine failure needs separate evidence. At the inspected #69408 head, REST-specific submission and the `>= 1000` history guard are still present. The issue describes a planned narrowing of that PR; do not treat that narrowing as an implemented REST fix or close this follow-up on that basis. **Evidence still needed:** exact Doris commit, REST server implementation/version, effective Iceberg client version, sanitized catalog configuration, and the executable probe/metadata fixture. Capture metadata before cleanup, after cleanup, after the competing spec commit and after DROP, including UUID, schema IDs/field IDs, all retained specs, default spec and `last-partition-id`. Provide the actual HTTP requirements/updates, responses, ordered synchronization points and relevant FE/server logs, with credentials removed. If claiming reader impact, include engine/version, query and complete error separately. No query profile is needed to establish this commit race. **Recommended next steps:** - Exercise the public Doris DROP path against a real REST server, pausing between DROP staging and HTTP application. Cover both current-partitioned and now-unpartitioned tables after unused-history cleanup, including the explicit low IDs and unchanged counter 999. - Define a server-enforced generation precondition bound to the metadata Doris validated, or equivalent atomic retained-source validation. Recheck it on every server retry and cover external writers. Standard 1.11.0 requirements alone cannot express a complete retained-spec comparison; a server extension needs an explicit capability/compatibility contract. - Verify that the race is rejected while unrelated legal drops, nested drops, ordinary SDK authentication and cache refresh still work. Test both commit orders and conflict retries. Any temporary restriction must fail explicitly, document its scope and avoid counter-threshold proofs, client-only locks or an unguarded fallback. Breakwater-GitHub-Analysis-Slot: slot_67c7ebb2d969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
