Hi Stephan,
I am facing S3 consistency related issue with the exception pasted at the
end:
We were able to solve the s3 sync issue by adding System.currentTime to
inprogressPrefix, inprogressSuffix, s3PendingPrefix and s3PendingSuffix
properties of BucketingSink.
I tried another approach by
Hi Stephan,
I am facing S3 consistency related issue with the exception pasted at the
end:
We were able to solve the s3 sync issue by adding System.currentTime to
inprogressPrefix, inprogressSuffix, s3PendingPrefix and s3PendingSuffix
properties of BucketingSink.
I tried another approach by
It is very important to point out that the Bucketing sink can currently NOT
work properly on S3. It assumes a consistent file system (like listing /
renaming works consistently), and S3 is only eventually consistent. I
assume that this eventual consistency of S3 is the cause of your error.
There
Hi Aneesha,
the logs would show that Flink is going through a recovery cycle. Recovery
means to cancel running tasks and start them again.
If you don't see something like that in the logs, Flink continues to
processing.
I'm not familiar with the details of S3, so I can't tell if the exception
Hello,
I am using flink 1.2 and writing records to S3 using rolling sink.
I am encountering this S3 write error quite frequently :
TimerException{com.amazonaws.services.s3.model.AmazonS3Exception: Status Code:
404, AWS Service: Amazon S3, AWS Request ID: B573887B1850BF28, AWS Error Code:
