deepakpanda93 commented on issue #9244: URL: https://github.com/apache/hudi/issues/9244#issuecomment-4845368525
Closing this out — it's a Flink write-failure-recovery scenario, and the reader-side gap it exposed is addressed in 1.x. What happened: the datanode downtime killed the Flink write mid-flight and left an uncommitted / un-rolled-back log block in the bucket file group; after restart, that leftover block was read alongside the retried write, producing duplicates that carry the failed write's hoodie meta fields. In 1.x this is handled: - the log reader now skips uncommitted log files/blocks (HUDI-6952, [#9879](https://github.com/apache/hudi/pull/9879), in 1.0.0+), so a failed write's leftover block is no longer read; - the completion-time-based timeline (1.0) means readers only surface data from completed instants; - Flink failover handling was hardened (partial failover [#7208], plus fixes to re-trigger failover after pending instants are recommitted). The original report was on 0.13.0, which predates these. Since the timeline for the failed run was never shared and the relevant recovery/reader code has been substantially reworked, I'm closing this. If you can still reproduce duplicates after a write failure on a recent 1.x release (Flink + bucket index + MOR), please reopen with the .hoodie/ timeline listing around the failure and we'll dig in. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
