deepakpanda93 commented on issue #16108: URL: https://github.com/apache/hudi/issues/16108#issuecomment-4892554188
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: This appears to be fixed. Idempotent Spark streaming writes keyed by batch id were implemented in PR #6098 (HUDI-4389, "Make HoodieStreamingSink idempotent") and hardened by #7383 / #7783 / #8558. HoodieStreamingSink now records the batch id as `_hudi_streaming_sink_checkpoint` in commit metadata and skips any already-committed micro-batch via canSkipBatch() (checking latestCommittedBatchId >= incomingBatchId per streaming checkpoint identifier), giving exactly-once micro-batch processing — see HoodieStreamingSink.scala (canSkipBatch / SINK_CHECKPOINT_KEY). This has been in place since release-0.13.0. If you're still seeing duplication with Spark streaming writes on a recent version, please reopen with repro steps (Hudi version, write config, and how batches are replayed). Closing as fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
