[
https://issues.apache.org/jira/browse/FLINK-41057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41057:
-----------------------------------
Labels: pull-request-available (was: )
> Iceberg append-only tables incorrectly append before-images from UPDATE and
> DELETE events
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-41057
> URL: https://issues.apache.org/jira/browse/FLINK-41057
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.7.0
> Reporter: Yanquan Lv
> Assignee: Yanquan Lv
> Priority: Critical
> Labels: pull-request-available
>
> h2. BACKGROUND
> The Iceberg sink must handle CDC events according to the target table's write
> capabilities.
> For tables with identifier fields, UPDATE events require deleting the
> before-image and inserting the after-image. For append-only tables without
> identifier fields, UPDATE events should only append the after-image, and
> DELETE events should be ignored.
> The fix for FLINK-40640 introduced unconditional conversion of UPDATE events
> into DELETE(before) followed by INSERT(after), without distinguishing these
> two types of target tables.
> h2. PROBLEM
> Append-only writers do not apply deletion semantics to rows marked as DELETE.
> Instead, they append those rows as ordinary records.
> As a result:
> - UPDATE events append both the before-image and the after-image,
> introducing an extra copy of the old record.
> - DELETE events append the deleted record instead of leaving the target
> table unchanged.
> The UPDATE behavior is a regression introduced by the unconditional UPDATE
> splitting. The DELETE behavior is a pre-existing issue.
> h2. REPRODUCTION SCENARIO
> Use an Iceberg target table without identifier fields.
> 1. Insert a record: (1, "old").
> 2. Update the record from (1, "old") to (1, "new").
> 3. Commit the changes.
> Actual table contents:
> - (1, "old")
> - (1, "old")
> - (1, "new")
> Expected table contents:
> - (1, "old")
> - (1, "new")
> If a subsequent DELETE event for (1, "new") is received, the table contents
> should remain unchanged. No additional record should be appended.
> h2. EXPECTED BEHAVIOR
> For append-only target tables:
> - INSERT and REPLACE: append the after-image.
> - UPDATE: append only the after-image.
> - DELETE: ignore the event.
> For target tables with identifier fields:
> - Preserve existing equality-delete behavior.
> - Keep partition-changing UPDATE events working correctly.
> Determine the write capability from the actual target Iceberg table's
> identifier fields, rather than solely from primary keys declared in the
> incoming CDC schema.
> Retaining historical records in an append-only table is intentional. This
> issue does not introduce upsert or delete support for such tables.
> h2. ACCEPTANCE CRITERIA
> - Cover partitioned and unpartitioned append-only tables.
> - Cover mixed target-table capabilities within the same writer.
> - Cover mismatches between incoming CDC primary keys and target-table
> identifier fields.
> - Preserve correct UPDATE and DELETE behavior for tables with identifier
> fields.
> - Correctly invalidate cached writer factories on schema changes, including
> when an ignored DELETE has not created a task writer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
