wangyum opened a new issue, #17314: URL: https://github.com/apache/iceberg/issues/17314
### Apache Iceberg version 1.11.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 `DynamicWriteResultSerializer` was modified in #14810 to add a `specId` field to the serialized format, but `getVersion()` was not bumped from `1`. This breaks the `SimpleVersionedSerializer` versioning contract: if a checkpoint from a job running the pre-#14810 code (V1, no `specId`) is restored on the current code (still claiming V1 but expecting `specId`), the `readInt()` for `specId` will consume the first four bytes of the `WriteResult` payload, producing a corrupted `WriteResult` and silent data loss. ### Steps to Reproduce 1. Start a Flink job using the dynamic Iceberg sink on a version before #14810 (serializes `WriteTarget` + `WriteResult` bytes as V1). 2. Take a checkpoint. 3. Upgrade the job to a version with #14810 (serializes `TableKey` + `specId` + `WriteResult` bytes, but still claims V1). 4. Restore from the checkpoint. 5. `deserialize(1, ...)` runs the new V1 code path, which calls `TableKey.deserializeFrom(view)` (reads only `tableName` + `branch`) then `view.readInt()` for `specId` — but the old V1 bytes have `schemaId` (an int) immediately after `branch`, so `readInt()` consumes `schemaId` as `specId`, and the remaining bytes (starting from the real `specId` field) are passed to `WriteResultSerializer.deserialize()`, producing a corrupt `WriteResult`. ### Expected Behavior The serializer should use distinct version numbers for the two formats so that old V1 bytes are deserialized using the old V1 layout (`WriteTarget` + `WriteResult`) and new V2 bytes are deserialized using the new V2 layout (`TableKey` + `specId` + `WriteResult`). ### Actual Behavior Both formats claim version 1, so the deserializer cannot distinguish them, leading to silent corruption on upgrade. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
