t3hw commented on code in PR #14797: URL: https://github.com/apache/iceberg/pull/14797#discussion_r2679344032
########## docs/docs/kafka-connect.md: ########## @@ -364,6 +368,39 @@ See above for creating two tables. } ``` +### Change data capture +This example applies inserts, updates, and deletes based on the value of a field in the record. +For example, if the `_cdc_op` field is set to `I` or `R` then the record is inserted, if `U` then it is +upserted, and if `D` then it is deleted. This requires that the table be in Iceberg v2 format. +The Iceberg identifier field(s) are used to identify a row, if that is not set for the table, +then the `iceberg.tables.default-id-columns` or `iceberg.table.\<table name\>.id-columns`configuration +can be set instead. CDC can be combined with multi-table fan-out. The property `iceberg.tables.use-dv` +can be set to `false` to disable delete vector (DV) mode for compatibility with v2 tables. Note that DV mode Review Comment: @hladush Yeah, you're right that it's breaking on V2 tables. See this comment for context: https://github.com/apache/iceberg/pull/14797#issuecomment-3645891074 I'm hesitant to just flip it to false though. The whole reason I enabled DVs in the first place was that non-DV position deletes can really hurt query performance when you have a lot of deletes. Better approach would be to check the table version first - use DV deletes for V3, fall back to non-DV for V2. I need to look into how we can query the catalog for the format version during connector setup. For now the error message at least makes it pretty obvious what's wrong, so users can either upgrade to V3 or manually disable DV support if they're stuck on V2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
