Thanks for the suggestion, Yuan.
How about naming the newly in-memory state-backend, which supports incremental checkpoint, as HeapStateBackend . And let the default state-backend still stay as HashMapStateBackend. By doing so, we can: 1) the default value of parameter state.backend.incremental could be set as true safely, which is so widely accepted according to the replies in this thread. 2) your newly in-memory state-backend could also be merged and users who want to try in-memory incremental checkpoints could also have a solution. We can also set the newly HeapStateBackend as default state-backend in the future if the feature of incremental checkpoint is stable. What do you think? Best, Lihe Ma 在 2022-06-15 21:03:31，"Feifan Wang" <zoltar9...@163.com> 写道： >Thanks for bringing this up. >Strongly +1 > > > > >—————————————— >Name: Feifan Wang >Email: zoltar9...@163.com > > >---- Replied Message ---- >| From | Yuan Mei<yuanmei.w...@gmail.com> | >| Date | 06/15/2022 11:41 | >| To | dev<dev@flink.apache.org> , ><ro...@ververica.com> | >| Subject | Re: [DISCUSS ] Make state.backend.incremental as true by default | >Thanks for bringing this up. > >I am +1 on making incremental checkpoints by default for RocksDB, but not >universally for all state backends. > >Besides being widely used in prod, enabling incremental checkpoint for >RocksDB by default is also a pre-requisite when enabling task-local by >default FLINK-15507 <https://issues.apache.org/jira/browse/FLINK-15507> > >The incremental checkpoint for the hashmap statebackend is under review >right now. CC @ro...@ververica.com <ro...@ververica.com> , which is not a >good idea being enabled by default in the first version. > >Best, > >Yuan > >On Tue, Jun 14, 2022 at 7:33 PM Jiangang Liu <liujiangangp...@gmail.com> >wrote: > >+1 for the suggestion. We have use the incremental checkpoint in our >production for a long time. > >Hangxiang Yu <master...@gmail.com> 于2022年6月14日周二 15:41写道： > >+1 >It's basically enabled in most scenarios in production environments. >For HashMapStateBackend, it will adopt a full checkpoint even if we >enable >incremental checkpoint. It will also support incremental checkpoint after >[1]. It's compatible. >BTW, I think we may also need to improve the documentation of incremental >checkpoints which users usually ask. There are some tickets like [2][3]. > >Best, >Hangxiang. > >[1] https://issues.apache.org/jira/browse/FLINK-21648 >[2] https://issues.apache.org/jira/browse/FLINK-22797 >[3] https://issues.apache.org/jira/browse/FLINK-7449 > >On Mon, Jun 13, 2022 at 7:48 PM Rui Fan <1996fan...@gmail.com> wrote: > >Strongly +1 > >Best, >Rui Fan > >On Mon, Jun 13, 2022 at 7:35 PM Martijn Visser < >martijnvis...@apache.org > >wrote: > >BTW, from my knowledge, nothing would happen for >HashMapStateBackend, >which does not support incremental checkpoint yet, when enabling >incremental checkpoints. > >Thanks Yun, if no errors would occur then definitely +1 to enable it >by >default > >Op ma 13 jun. 2022 om 12:42 schreef Alexander Fedulov < >alexan...@ververica.com>: > >+1 > >From my experience, it is actually hard to come up with use cases >where >incremental checkpoints should explicitly not be enabled with the >RocksDB >state backend. If the state is so small that the full snapshots do >not >have any negative impact, one should consider using >HashMapStateBackend >anyway. > >Best, >Alexander Fedulov > > >On Mon, Jun 13, 2022 at 12:26 PM Jing Ge <j...@ververica.com> >wrote: > >+1 > >Glad to see the kickoff of this discussion. Thanks Lihe for >driving >this! > >We have actually already discussed it internally a few months >ago. >After >considering some corner cases, all agreed on enabling the >incremental >checkpoint as default. > >Best regards, >Jing > >On Mon, Jun 13, 2022 at 12:17 PM Yun Tang <myas...@live.com> >wrote: > >Strongly +1 for making incremental checkpoints as default. Many >users >have >ever been asking why this configuration is not enabled by >default. > >BTW, from my knowledge, nothing would happen for >HashMapStateBackend, >which does not support incremental checkpoint yet, when >enabling >incremental checkpoints. > > >Best >Yun Tang >________________________________ >From: Martijn Visser <martijnvis...@apache.org> >Sent: Monday, June 13, 2022 18:05 >To: dev@flink.apache.org <dev@flink.apache.org> >Subject: Re: [DISCUSS ] Make state.backend.incremental as true >by >default > >Hi Lihe, > >What happens if we enable incremental checkpoints by default >while >the >used >memory backend is HashMapStateBackend, which doesn't support >incremental >checkpoints? > >Best regards, > >Martijn > >Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma <ma_l...@163.com>: > >Hi, Everyone, > >I would like to open a discussion on setting incremental >checkpoint >as >default behavior. > >Currently, the configuration of state.backend.incremental is >set >as >false >by default. Incremental checkpoint has been adopted widely in >industry >community for many years , and it is also well-tested from >the >feedback >in >the community discussion. Incremental checkpointing is more >light-weighted: >shorter checkpoint duration, less uploaded data and less >resource >consumption. > >In terms of backward compatibility, enable incremental >checkpointing >would >not make any data loss no matter restoring from a full >checkpoint/savepoint >or an incremental checkpoint. > >FLIP-193 (Snapshot ownership)[1] has been released in 1.15, >incremental >checkpoint no longer depends on a previous restored >checkpoint >in >default >NO_CLAIM mode, which makes the checkpoint lineage much >cleaner, >it >is a >good chance to change the configuration >state.backend.incremental >to >true >as default. > >Thus, based on the above discussion, I suggest to make >state.backend.incremental as true by default. What do you >think >of >this >proposal? > >[1] > > > > > > > >https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership > >Best regards, >Lihe Ma > > > > > > > >
