No, I think what he means is to trigger the checkpoint at slightly different times at the different sources so the different parts of the pipeline would not checkpoint at the same time.
Gyula On Wed, Sep 13, 2023 at 10:32 AM Hangxiang Yu <master...@gmail.com> wrote: > Hi, Matyas. > Do you mean something like adjusting checkpoint intervals dynamically > or frequency of uploading files according to the pressure of the durable > storage ? > > On Wed, Sep 13, 2023 at 9:12 AM Őrhidi Mátyás <matyas.orh...@gmail.com> > wrote: > >> Hey folks, >> >> Is it possible to add some sort of jitter to the checkpointing logic for >> massively parallel jobs to mitigate the burst impact on the durable storage >> when a checkpoint is triggered? >> >> Thanks, >> Matyas >> > > > -- > Best, > Hangxiang. >
