a8356555 opened a new issue, #11573:
URL: https://github.com/apache/iceberg/issues/11573
### Apache Iceberg version
1.5.2
### Query engine
Athena
### Please describe the bug π
Hi,
I'm using MySQL Flink CDC with Iceberg 1.5.2 and Flink 1.16. I have a table
partitioned by the status column, but this column is subject to updates. When
an update occurs, I encounter duplicate records in the Iceberg table, which is
not the desired behavior.
Is there a way to properly handle updates on a partition column in Iceberg
to avoid duplicates?
here is the sql of my flink CDC
```
CREATE CATALOG glue_catalog WITH (
'type'='iceberg',
'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
'warehouse'='s3://my-bucket'
);
CREATE TABLE mysql_cdc_source
(
id INT,
status INT,
value INT,
...,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'mysql-cdc',
'hostname' = 'mysql',
'port' = '3306',
'username' = 'XXX',
'password' = 'XXX',
'database-name' = 'XXX',
'table-name' = 'XXX',
'debezium.snapshot.mode' = 'when_needed',
'server-id' = '1'
);
CREATE TABLE IF NOT EXISTS glue_catalog.my_db.my_table(
`id` INT NOT NULL,
`status` INT,
`value` INT,
...,
iceberg)
PRIMARY KEY (id,status) NOT ENFORCED
) PARTITIONED BY (
status
) WITH (
'format-version'='2',
'write.metadata.delete-after-commit.enabled'='true',
'write.upsert.enabled'='true',
'write.delete.mode'='merge-on-read',
'write.merge.mode'='merge-on-read',
'write.update.mode'='merge-on-read'
);
INSERT INTO glue_catalog.my_db.my_table
SELECT
*
FROM mysql_cdc_source;
```
data before updating mysql:
data after updating mysql (duplicated row showed up):
get same results using spark 3.4:
<img width="144" alt="image"
src="https://github.com/user-attachments/assets/78b28982-ebf0-4a53-9bf6-248060b847af";>
query `glue_catalog.my_db.my_table.files` using spark 3.4
Thanks!
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
