ad1happy2go commented on issue #19593: URL: https://github.com/apache/hudi/issues/19593#issuecomment-5539056754
Thanks for the deep investigation. One correction first: `ROLLBACK_USING_MARKERS_ENABLE=false` and `metadata.enable=false` are not why the duplicates survived. Hudi cleans up leftover files from retried tasks on every commit regardless of those settings, and failed-write cleaning is on by default. So something slipped past the safety nets. **1. Same record in two file groups, one commit (your biggest cluster)** Why: either a Spark stage recompute wrote the same rows twice and both files got committed as valid, or the duplicate file's marker was lost so cleanup never saw it. Fix: check the origination commit's `.deltacommit`. If both file ids are listed there, it's the recompute case and you need to reduce task/stage retries. If only one, a marker was lost, so set `hoodie.write.markers.type=DIRECT`. **2. Keys re-inserted into a neighbor file group** Why: the index only checks base files, so records still in log files waiting for the 6-hour compaction are invisible and get inserted again. Fix: compact more often. The record index would fix this fully, but it's global and your same-guid-in-two-collections requirement rules it out. **3. Whole batch written twice, minutes apart** Why: app-side. The retry block re-runs `.save()` when the metrics call after it throws, and the Kinesis iterator reset replays committed batches. Fix: move metrics out of the retry, and store the batch sequenceId in commit metadata so a replayed batch can check it and skip. **4. PROMOTE leaves the old copy behind** Fix: send an explicit delete to the old collection on PROMOTE. The partition-scoped index can't do it for you. **5. Soft deletes doing nothing** Fix: `operationTime=0L` always loses the precombine. Use the real timestamp. **6. Existing ~720 duplicates** They never self-heal. One-time cleanup: rewrite affected partitions keeping the newest copy per guid, or CLI `repair deduplicate`. Can you post the writeStats check from #1 plus your current `hoodie.write.markers.type`? That settles the main mechanism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
