Re: How to monitor changes in the existing files using flink 1.17.2

2024-01-09 Thread Yu Chen
Hi Nitin, In the Flink file system connector, a collection of file paths is used by flink to identify whether a file has been processed or not in the state[1]. So, if your file path has not been updated but the content has been updated, it will not be reprocessed in that case. Meanwhile, here

How to monitor changes in the existing files using flink 1.17.2

2024-01-09 Thread Nitin Saini
Hi Flink Community, I was using flink 1.12.7 readFile to read files from the s3 it was able to monitor if there are new files added or updation in the existing files as well. But now I have migrated to flink 1.17.2 and using FileSource to read files from s3 it was able to monitor if new files are