[
https://issues.apache.org/jira/browse/FLINK-41057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanquan Lv updated FLINK-41057:
-------------------------------
Description:
h2. BACKGROUND
The Iceberg sink must handle CDC events according to the target table's write
capabilities.
For tables with identifier fields, UPDATE events require deleting the
before-image and inserting the after-image. For append-only tables without
identifier fields, UPDATE events should only append the after-image, and DELETE
events should be ignored.
The fix for FLINK-40640 introduced unconditional conversion of UPDATE events
into DELETE(before) followed by INSERT(after), without distinguishing these two
types of target tables.
h2. PROBLEM
Append-only writers do not apply deletion semantics to rows marked as DELETE.
Instead, they append those rows as ordinary records.
As a result:
- UPDATE events append both the before-image and the after-image, introducing
an extra copy of the old record.
- DELETE events append the deleted record instead of leaving the target table
unchanged.
The UPDATE behavior is a regression introduced by the unconditional UPDATE
splitting. The DELETE behavior is a pre-existing issue.
h2. REPRODUCTION SCENARIO
Use an Iceberg target table without identifier fields.
1. Insert a record: (1, "old").
2. Update the record from (1, "old") to (1, "new").
3. Commit the changes.
Actual table contents:
- (1, "old")
- (1, "old")
- (1, "new")
Expected table contents:
- (1, "old")
- (1, "new")
If a subsequent DELETE event for (1, "new") is received, the table contents
should remain unchanged. No additional record should be appended.
h2. EXPECTED BEHAVIOR
For append-only target tables:
- INSERT and REPLACE: append the after-image.
- UPDATE: append only the after-image.
- DELETE: ignore the event.
For target tables with identifier fields:
- Preserve existing equality-delete behavior.
- Keep partition-changing UPDATE events working correctly.
Determine the write capability from the actual target Iceberg table's
identifier fields, rather than solely from primary keys declared in the
incoming CDC schema.
Retaining historical records in an append-only table is intentional. This issue
does not introduce upsert or delete support for such tables.
h2. ACCEPTANCE CRITERIA
- Cover partitioned and unpartitioned append-only tables.
- Cover mixed target-table capabilities within the same writer.
- Cover mismatches between incoming CDC primary keys and target-table
identifier fields.
- Preserve correct UPDATE and DELETE behavior for tables with identifier
fields.
- Correctly invalidate cached writer factories on schema changes, including
when an ignored DELETE has not created a task writer.
was:
h2. BACKGROUND
The Iceberg sink must handle CDC events according to the target table's write
capabilities.
For tables with identifier fields, UPDATE events require deleting the
before-image and inserting the after-image. For append-only tables without
identifier fields, UPDATE events should only append the after-image, and DELETE
events should be ignored.
The fix for FLINK-40640 introduced unconditional conversion of UPDATE events
into DELETE(before) followed by INSERT(after), without distinguishing these two
types of target tables.
h2.
PROBLEM
Append-only writers do not apply deletion semantics to rows marked as DELETE.
Instead, they append those rows as ordinary records.
As a result:
- UPDATE events append both the before-image and the after-image, introducing
an extra copy of the old record.
- DELETE events append the deleted record instead of leaving the target table
unchanged.
The UPDATE behavior is a regression introduced by the unconditional UPDATE
splitting. The DELETE behavior is a pre-existing issue.
h2.
REPRODUCTION SCENARIO
Use an Iceberg target table without identifier fields.
1. Insert a record: (1, "old").
2. Update the record from (1, "old") to (1, "new").
3. Commit the changes.
Actual table contents:
- (1, "old")
- (1, "old")
- (1, "new")
Expected table contents:
- (1, "old")
- (1, "new")
If a subsequent DELETE event for (1, "new") is received, the table contents
should remain unchanged. No additional record should be appended.
h2.
EXPECTED BEHAVIOR
For append-only target tables:
- INSERT and REPLACE: append the after-image.
- UPDATE: append only the after-image.
- DELETE: ignore the event.
For target tables with identifier fields:
- Preserve existing equality-delete behavior.
- Keep partition-changing UPDATE events working correctly.
Determine the write capability from the actual target Iceberg table's
identifier fields, rather than solely from primary keys declared in the
incoming CDC schema.
Retaining historical records in an append-only table is intentional. This issue
does not introduce upsert or delete support for such tables.
h2.
ACCEPTANCE CRITERIA
- Cover partitioned and unpartitioned append-only tables.
- Cover mixed target-table capabilities within the same writer.
- Cover mismatches between incoming CDC primary keys and target-table
identifier fields.
- Preserve correct UPDATE and DELETE behavior for tables with identifier fields.
- Correctly invalidate cached writer factories on schema changes, including
when an ignored DELETE has not created a task writer.
h2.
RELATED ISSUE
FLINK-40640
> Iceberg append-only tables incorrectly append before-images from UPDATE and
> DELETE events
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-41057
> URL: https://issues.apache.org/jira/browse/FLINK-41057
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.7.0
> Reporter: Yanquan Lv
> Assignee: Yanquan Lv
> Priority: Critical
>
> h2. BACKGROUND
> The Iceberg sink must handle CDC events according to the target table's write
> capabilities.
> For tables with identifier fields, UPDATE events require deleting the
> before-image and inserting the after-image. For append-only tables without
> identifier fields, UPDATE events should only append the after-image, and
> DELETE events should be ignored.
> The fix for FLINK-40640 introduced unconditional conversion of UPDATE events
> into DELETE(before) followed by INSERT(after), without distinguishing these
> two types of target tables.
> h2. PROBLEM
> Append-only writers do not apply deletion semantics to rows marked as DELETE.
> Instead, they append those rows as ordinary records.
> As a result:
> - UPDATE events append both the before-image and the after-image,
> introducing an extra copy of the old record.
> - DELETE events append the deleted record instead of leaving the target
> table unchanged.
> The UPDATE behavior is a regression introduced by the unconditional UPDATE
> splitting. The DELETE behavior is a pre-existing issue.
> h2. REPRODUCTION SCENARIO
> Use an Iceberg target table without identifier fields.
> 1. Insert a record: (1, "old").
> 2. Update the record from (1, "old") to (1, "new").
> 3. Commit the changes.
> Actual table contents:
> - (1, "old")
> - (1, "old")
> - (1, "new")
> Expected table contents:
> - (1, "old")
> - (1, "new")
> If a subsequent DELETE event for (1, "new") is received, the table contents
> should remain unchanged. No additional record should be appended.
> h2. EXPECTED BEHAVIOR
> For append-only target tables:
> - INSERT and REPLACE: append the after-image.
> - UPDATE: append only the after-image.
> - DELETE: ignore the event.
> For target tables with identifier fields:
> - Preserve existing equality-delete behavior.
> - Keep partition-changing UPDATE events working correctly.
> Determine the write capability from the actual target Iceberg table's
> identifier fields, rather than solely from primary keys declared in the
> incoming CDC schema.
> Retaining historical records in an append-only table is intentional. This
> issue does not introduce upsert or delete support for such tables.
> h2. ACCEPTANCE CRITERIA
> - Cover partitioned and unpartitioned append-only tables.
> - Cover mixed target-table capabilities within the same writer.
> - Cover mismatches between incoming CDC primary keys and target-table
> identifier fields.
> - Preserve correct UPDATE and DELETE behavior for tables with identifier
> fields.
> - Correctly invalidate cached writer factories on schema changes, including
> when an ignored DELETE has not created a task writer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
