Thank you Taher, We are not on EMR but great to know that s3 and streaming sink should be working fine based on your explanation.
On Sun, Aug 18, 2019 at 8:23 AM taher koitawala <taher...@gmail.com> wrote: > Hi Swapnil, > We faced this problem once, I think changing checkpoint dir to hdfs > and keeping sink dir to s3 with EMRFS s3 consistency enabled solves this > problem. If you are not using emr then I don't know how else it can be > solved. But in a nutshell because EMRFS s3 consistency uses Dynamo DB in > the back end to check for all files being written to s3. It kind of makes > s3 consistent and Streaming file sink works just fine. > > > > On Sat, Aug 17, 2019, 3:32 AM Swapnil Kumar <swku...@zendesk.com> wrote: > >> Hello, We are using Flink to process input events and aggregate and write >> o/p of our streaming job to S3 using StreamingFileSink but whenever we try >> to restore the job from a savepoint, the restoration fails with missing >> part files error. As per my understanding, s3 deletes those >> part(intermittent) files and can no longer be found on s3. Is there a >> workaround for this, so that we can use s3 as a sink? >> >> -- >> Thanks, >> Swapnil Kumar >> > -- Thanks, Swapnil Kumar
