Hi Averell, Happy to hear that the problem is no longer there and if you have more news from your debugging, let us know.
The thing that I wanted to mention is that from what you are describing, the problem does not seem to be related to checkpointing, but to the fact that applying your filter on the 100’s of thousands of small files takes time. This may help with your debugging. Cheers, Kostas > On Sep 24, 2018, at 2:10 AM, Averell <lvhu...@gmail.com> wrote: > > Hi Vino, and all, > > I tried to avoid the step to get File Status, and found that the problem is > not there any more. I guess doing that with every single file out of 100K+ > files on S3 caused some issue with checkpointing. > Still trying to find the cause, but with lower priority now. > > Thanks for your help. > > Regards, > Averell > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
