GitHub user danny0405 edited a comment on the discussion: RLI support for Flink streaming
In general, the file creation must be in the same lifecycle of the Flink checkpoint, otherwise, Flink might deem the checkpoint as successful while Hudi actually fails in commit, this inconsistency will make it hard for job recover because checkpoint is what Flink recognize to recover from. > I suppose it might be possible to utilize Flink's local state on task > managers, and flush buffers there, instead of writing to remote storage > during the checkpoint. @geserdugarov are you suggesting to write to local disk first then copy the files into remote? can you elaborate a little more? GitHub link: https://github.com/apache/hudi/discussions/17452#discussioncomment-15205674 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
