t; > >> for
>>> > execution.savepoint-restore-mode/execution.recovery.claim-mode.
>>> > > > > >> 3. Could the local checkpoint be cleaned
>>> > > > > >> if execution.checkpointing.local-copy.enabled is true and
>>> > > > > >> execution.recovery.from-local is false ? I fo
> > >> basically default configuration in our production environment.
>> > > > > >>
>> > > > > >>
>> > > > > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan <
>> zakelly@gmail.com
>> > >
>
> >
> > > > > >> > 1. We shall not describe the configuration with its
> > > implementation
> > > > > for
> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> > > > >
> > >> > state-backend,
> > > > >> > > it would write two streams and for Rocksdb state-backend, it
> > would
> > > > use
> > > > >> > > hard-link for backup. Thus, I think
> > > > >> > > 'exec
> > >> > the implementation, thus I name it as a copy of the checkpoint in
> the
> > > >> > local disk, regardless of the way of generating it. The word
> 'backup'
> > > is
> > > >> > also suitable for this case, so I a
>> > 'execution.checkpointing.local-backup.*' if no one objects.
> > >> >
> > >> > 2. What does the 'execution.checkpointing.data-inline-threshold'
> > >> mean? It
> > >> > > seems not so easy to understand.
&
for the size threshold
> below
> >> > which state chunks will store inline with the metadata, thus I call it
> >> > 'data-inline-threshold'.
> >> >
> >> >
> >> > Best,
> >> > Zakelly
> >> >
> >> &
Zakelly
>> >
>> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang wrote:
>> >
>> > > Hi Zakelly,
>> > >
>> > > Thanks for driving this topic. I have two concerns here:
>> > >
>> > > 1. We shall not describe the configuration with its imp
.checkpointing.local-copy.*' options, for hashmap
> > state-backend,
> > > it would write two streams and for Rocksdb state-backend, it would use
> > > hard-link for backup. Thus, I think
> > > 'execution.checkpointing.local-backup.*' looks better.
>
'execution.checkpointing.data-inline-threshold' mean?
> > It seems not so easy to understand.
> >
> > Best
> > Yun Tang
> >
> > From: Piotr Nowojski
> > Sent: Thursday, January 4, 2024 22:37
> > To: dev@flink
st
> Yun Tang
> ____
> From: Piotr Nowojski
> Sent: Thursday, January 4, 2024 22:37
> To: dev@flink.apache.org
> Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
> Recovery Configuration
>
> Hi,
>
> Thanks fo
ski
Sent: Thursday, January 4, 2024 22:37
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery
Configuration
Hi,
Thanks for trying to clean this up! I don't have strong opinions on the
topics discussed here, so generally speaking +1 from
Hi,
Thanks for trying to clean this up! I don't have strong opinions on the
topics discussed here, so generally speaking +1 from my side!
Best,
Piotrek
śr., 3 sty 2024 o 04:16 Rui Fan <1996fan...@gmail.com> napisał(a):
> Thanks for the feedback!
>
> Using the `execution.checkpointing.incrementa
Thanks for the feedback!
Using the `execution.checkpointing.incremental.enabled`,
and enabling it by default sounds good to me.
Best,
Rui
On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan wrote:
> Hi Rui,
>
> Thanks for your comments!
>
> IMO, given that the state backend can be plugably loaded (as
Hi Rui,
Thanks for your comments!
IMO, given that the state backend can be plugably loaded (as you can
specify a state backend factory), I prefer not providing state backend
specified options in the framework.
Secondly, the incremental checkpoint is actually a sharing file strategy
across checkp
Hi Zakelly,
I'm not sure whether we could add the state backend type in the
new key name of state.backend.incremental. It means we use
`execution.checkpointing.rocksdb-incremental` or
`execution.checkpointing.rocksdb-incremental.enabled`.
So far, state.backend.incremental only works for rocksdb s
Hi everyone,
Thanks all for your comments!
As many of you have questions about the names for boolean options, I
suggest we make a naming rule for them. For now I could think of three
options:
Option 1: Use enumeration options if possible. But this may cause some name
collisions or confusion as w
Hi Zakelly,
Thanks for driving this! The organization of the configuration option
in the FLIP looks much cleaner and easier to understand. +1 to the
FLIP.
Just some questions from me.
1. I think the change to the ConfigOptions should be put in the
`Public Interface` section, instead of `Proposed
Hi Zakelly,
> Considering the name occupation, how about naming it as
> `execution.checkpointing.type`?
`Checkpoint Type`[1,2] is used to describe aligned/unaligned
checkpoint, I am inclined to make a choice between
`execution.checkpointing.incremental` and
`execution.checkpointing.incremental.e
Hi Lijie,
Thanks for the reminder! I missed this.
Considering the name occupation, how about naming it as
`execution.checkpointing.type`?
Actually I think the current `execution.checkpointing.mode` is confusing in
some ways, maybe `execution.checkpointing.data-consistency` is better.
Best,
Zak
Hi Zakelly,
>> I'm wondering if `execution.checkpointing.savepoint-dir` would be better.
`execution.checkpointing.dir` and `execution.checkpointing.savepoint-dir`
are also fine for me.
>> So I think an enumeration option `execution.checkpointing.mode` which
can be 'full' (default) or 'incrementa
Hi everyone,
Thanks all for your comments!
@Yanfei
> 1. For some state backends that do not support incremental checkpoint,
> how does the execution.checkpointing.incrementaloption take effect? Or
> is it better to put incremental under state.backend.xxx.incremental?
>
I'd rather not put the opt
Hi Zakelly,
Thanks for driving the discussion.
1.
>> But I'm not so sure since there is only one savepoint-related option.
Maybe someone else could share some thoughts here.
How about
state.savepoints.dir -> execution.checkpointing.savepoint.dir
state.checkpoints.dir -> execution.checkpointing.c
Hi Zakelly,
Thank you for creating the FLIP and starting the discussion.
The current arrangement of these options is indeed somewhat haphazard,
and the new arrangement looks much better. I have some questions about
the arrangement of some new configuration options:
1. For some state backends tha
Hi Rui Fan and Junrui,
Thanks for the reminder! I agree to change the
'execution.checkpointing.local-copy' to
'execution.checkpointing.local-copy.enabled'.
And for other suggestions Rui proposed:
1. How about execution.checkpointing.storage.type instead
> of execution.checkpointing.storage?
Ah
Hi Zakelly,
Thanks for driving this. I agree that the proposed restructuring of the
configuration options is largely positive. It will make understanding and
working with Flink configurations more intuitive.
Most of the proposed changes look great. Just a heads-up, as Rui Fan
mentioned, Flink cur
Hi Zakelly,
Thank you for driving this proposal!
Overall good for me. I have some questions about these names.
1. How about execution.checkpointing.storage.type instead of
execution.checkpointing.storage?
It's similar to state.backend.type.
2. How about execution.checkpointing.local-copy.enabl
Hi devs,
I'd like to start a discussion on FLIP-406: Reorganize State &
Checkpointing & Recovery Configuration[1].
Currently, the configuration options pertaining to checkpointing, recovery,
and state management are primarily grouped under the following prefixes:
- state.backend.* : configura
28 matches
Mail list logo
