StreamingFileSink question

2022-08-31 Thread David Clutter
I am using Flink 1.13.1 on AWS EMR 6.4. I have an existing application using DataStream API that I would like to modify to write output to S3. I am testing the StreamingFileSink with a bounded input. I have enabled checkpointing. A couple questions: 1) When the program finishes, all the files r

Re: StreamingFileSink question

2022-08-31 Thread David Anderson
If I remember correctly, there's a fix for this in Flink 1.14 (but the feature is disabled by default in 1.14, and enabled by default in 1.15). (I'm thinking that execution.checkpointing.checkpoints-after-tasks-finish.enabled [1] takes care of this.) With Flink 1.13 I believe you'll have to handle