deepakpanda93 commented on issue #8981: URL: https://github.com/apache/hudi/issues/8981#issuecomment-4845244137
This is fixed in Hudi 0.13.0+ (and all 1.x). Root cause: on 0.12.x, a composite record key was encoded differently by the two write paths — bulk_insert (row writer) produced values-only (value1,value2,value3) while upsert produced field1:value1,field2:value2,…. So the same logical key didn't match across bulk-insert-initialized and upserted rows, and the index treated them as distinct → duplicates. @ad1happy2go reproduced this on 0.12.2 and confirmed it's resolved in 0.13.0 (the 0.13.0 record/key-handling rework, RFC-46 / HUDI-3378; a composite-key + bulk-insert regression test was later added in HUDI-6953 [#10214](https://github.com/apache/hudi/pull/10214)). All 1.x releases include the fix. One important note: the bulk-insert that initializes the table and the subsequent upserts must run on the same fixed version so the key encoding is consistent. A table bootstrapped with an older bulk-insert (values-only keys) and then upserted on a newer version (field:value keys) will still mismatch — please re-initialize the table on 1.x and clean up any existing duplicates. Closing as fixed in 0.13.0+. If you still see this with composite keys on a recent 1.x release (table written end-to-end on that version), please reopen with a minimal repro. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
