https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html
You will find there a passage of the consistency model. Probably the system is putting them to the folder and Flink is triggered before they are consistent. What happens after Flink put s them on S3 ? Are they reused by another system ? Or is it just archival? If they are reused then probably go for a nosql solution (eg Dynamo), if they are just archived then use kinesis + s3 > On 24. Jul 2018, at 11:52, Averell <lvhu...@gmail.com> wrote: > > Could you please help explain more details on "/try read after write > consistency (assuming the files are not modified) /"? > I guess that the problem I got comes from the inconsistency in S3 files > listing. Otherwise, I would have got exceptions on file not found. > > My use case is to read output files from another system. That system was > built some years back, and is outputting files to their S3 bucket. There is > no file modification, only new files are being created. We want to avoid > modifying that system. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
