deepakpanda93 commented on issue #8474: URL: https://github.com/apache/hudi/issues/8474#issuecomment-4844850117
Closing this as stale — it was never reproduced (the requested duplicate-record meta fields were never shared, and it only manifested transiently on executor loss). That said, on the substance: your hypothesis is right — on a stage re-run after executor loss, a BLOOM-index tagLocation recompute can re-classify an update as an insert. Current Hudi mitigates this by persisting the tagged records before building the workload profile and writing (so the classification and the write use the same materialized tagging), and the broader "recompute on retry" robustness is being hardened ([#15017](https://github.com/apache/hudi/issues/15017), and retry test coverage in [#16717](https://github.com/apache/hudi/issues/16717) for 1.1.0). The most reliable way to avoid this class of issue is to use a deterministic, persisted index — the Record Level Index (hoodie.index.type=RECORD_INDEX, backed by the metadata table) — so tag lookups read stable index entries rather than recomputing a bloom scan that can vary across retries. If you can reproduce duplicate keys after executor loss on a recent 1.x release (ideally with RLI), please reopen with the _hoodie_record_key / _hoodie_file_name of the duplicates and the timeline, and we'll dig in. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
