Thanks for bringing this up. Strongly +1
—————————————— Name: Feifan Wang Email: zoltar9...@163.com ---- Replied Message ---- | From | Yuan Mei<yuanmei.w...@gmail.com> | | Date | 06/15/2022 11:41 | | To | dev<dev@flink.apache.org> , <ro...@ververica.com> | | Subject | Re: [DISCUSS ] Make state.backend.incremental as true by default | Thanks for bringing this up. I am +1 on making incremental checkpoints by default for RocksDB, but not universally for all state backends. Besides being widely used in prod, enabling incremental checkpoint for RocksDB by default is also a pre-requisite when enabling task-local by default FLINK-15507 <https://issues.apache.org/jira/browse/FLINK-15507> The incremental checkpoint for the hashmap statebackend is under review right now. CC @ro...@ververica.com <ro...@ververica.com> , which is not a good idea being enabled by default in the first version. Best, Yuan On Tue, Jun 14, 2022 at 7:33 PM Jiangang Liu <liujiangangp...@gmail.com> wrote: +1 for the suggestion. We have use the incremental checkpoint in our production for a long time. Hangxiang Yu <master...@gmail.com> 于2022年6月14日周二 15:41写道： +1 It's basically enabled in most scenarios in production environments. For HashMapStateBackend, it will adopt a full checkpoint even if we enable incremental checkpoint. It will also support incremental checkpoint after [1]. It's compatible. BTW, I think we may also need to improve the documentation of incremental checkpoints which users usually ask. There are some tickets like [2][3]. Best, Hangxiang. [1] https://issues.apache.org/jira/browse/FLINK-21648 [2] https://issues.apache.org/jira/browse/FLINK-22797 [3] https://issues.apache.org/jira/browse/FLINK-7449 On Mon, Jun 13, 2022 at 7:48 PM Rui Fan <1996fan...@gmail.com> wrote: Strongly +1 Best, Rui Fan On Mon, Jun 13, 2022 at 7:35 PM Martijn Visser < martijnvis...@apache.org wrote: BTW, from my knowledge, nothing would happen for HashMapStateBackend, which does not support incremental checkpoint yet, when enabling incremental checkpoints. Thanks Yun, if no errors would occur then definitely +1 to enable it by default Op ma 13 jun. 2022 om 12:42 schreef Alexander Fedulov < alexan...@ververica.com>: +1 From my experience, it is actually hard to come up with use cases where incremental checkpoints should explicitly not be enabled with the RocksDB state backend. If the state is so small that the full snapshots do not have any negative impact, one should consider using HashMapStateBackend anyway. Best, Alexander Fedulov On Mon, Jun 13, 2022 at 12:26 PM Jing Ge <j...@ververica.com> wrote: +1 Glad to see the kickoff of this discussion. Thanks Lihe for driving this! We have actually already discussed it internally a few months ago. After considering some corner cases, all agreed on enabling the incremental checkpoint as default. Best regards, Jing On Mon, Jun 13, 2022 at 12:17 PM Yun Tang <myas...@live.com> wrote: Strongly +1 for making incremental checkpoints as default. Many users have ever been asking why this configuration is not enabled by default. BTW, from my knowledge, nothing would happen for HashMapStateBackend, which does not support incremental checkpoint yet, when enabling incremental checkpoints. Best Yun Tang ________________________________ From: Martijn Visser <martijnvis...@apache.org> Sent: Monday, June 13, 2022 18:05 To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: [DISCUSS ] Make state.backend.incremental as true by default Hi Lihe, What happens if we enable incremental checkpoints by default while the used memory backend is HashMapStateBackend, which doesn't support incremental checkpoints? Best regards, Martijn Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma <ma_l...@163.com>: Hi, Everyone, I would like to open a discussion on setting incremental checkpoint as default behavior. Currently, the configuration of state.backend.incremental is set as false by default. Incremental checkpoint has been adopted widely in industry community for many years , and it is also well-tested from the feedback in the community discussion. Incremental checkpointing is more light-weighted: shorter checkpoint duration, less uploaded data and less resource consumption. In terms of backward compatibility, enable incremental checkpointing would not make any data loss no matter restoring from a full checkpoint/savepoint or an incremental checkpoint. FLIP-193 (Snapshot ownership)[1] has been released in 1.15, incremental checkpoint no longer depends on a previous restored checkpoint in default NO_CLAIM mode, which makes the checkpoint lineage much cleaner, it is a good chance to change the configuration state.backend.incremental to true as default. Thus, based on the above discussion, I suggest to make state.backend.incremental as true by default. What do you think of this proposal? [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership Best regards, Lihe Ma
