GitHub user HuangZhenQiu added a comment to the discussion: RocksDB as The Replica of MDT/RLI
@vinothchandar It is a known and resolved issue. Basically, even for append-only use cases, Flink ingestion could first finished checkpoint and later fail due the Hudi commit failure. In this case, job will fail and restart. How do we recover from this issue? We include Kafka offset into each of Hudi commit metadata. When job restart from failure, it will use the kafka offset in the last completed hudi commit to start rather than the kafka offset in flink checkpoint. We simply only use flink checkpoint mechanism for two phase commit protocol implementation not for any state management and recovery. Will the solution usable for Flink upsert? Yes, it will still work for Fink upsert, not matter we bootstrap record index or not, we don't relies on the state in Flink checkpoint to recovery. We may still use the Kafka offset in last completed Hudi commit to recover to guarantee exactly once data write to the target Hudi table GitHub link: https://github.com/apache/hudi/discussions/18296#discussioncomment-18390480 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
