Re: Incremental checkpoint branch

Hi! The concrete storage location (HDFS / S3) should be independent of the imeplementation. The design should work for both. Stephan On Mon, Mar 6, 2017 at 5:38 PM, Vishnu Viswanath < vishnu.viswanat...@gmail.com> wrote: > Thanks everyone for the reply. > > @Xiaogang design document looks

Re: Incremental checkpoint branch

Thanks everyone for the reply. @Xiaogang design document looks very good, looking forward to check the implementation and to try it out. I see that hdfs:// is mentioned as the backend storage in the documentation, what about s3://. Is it something that needed to be considered in the design, or is

Re: Incremental checkpoint branch

Hi Vinshnu, We have obtained an initial design of incremental checkpointing [1] and will start working on incremental checkpointing the next week. You can watch the issue FLINK-5053 [2] to get timely notification of the updates. All suggestions are welcome. [1]

Re: Incremental checkpoint branch

Vinshnu, You can find the latest design discussion for incremental checkpoint in http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-Flink-td15931.html @Stefan Richter

Incremental checkpoint branch

Hi, Can someone point me to the branch where the ongoing work for incremental checkpoint is going on, I would like to try it out even if the work is not complete. I have a use case where the state size increase about ~1gb every 5 minutes. Thanks, Vishnu