Correct, thanks for the clarification Gyula! On Wed, Sep 13, 2023 at 1:39 AM Gyula Fóra <gyula.f...@gmail.com> wrote:
> No, I think what he means is to trigger the checkpoint at slightly > different times at the different sources so the different parts of the > pipeline would not checkpoint at the same time. > > Gyula > > On Wed, Sep 13, 2023 at 10:32 AM Hangxiang Yu <master...@gmail.com> wrote: > >> Hi, Matyas. >> Do you mean something like adjusting checkpoint intervals dynamically >> or frequency of uploading files according to the pressure of the durable >> storage ? >> >> On Wed, Sep 13, 2023 at 9:12 AM Őrhidi Mátyás <matyas.orh...@gmail.com> >> wrote: >> >>> Hey folks, >>> >>> Is it possible to add some sort of jitter to the checkpointing logic for >>> massively parallel jobs to mitigate the burst impact on the durable storage >>> when a checkpoint is triggered? >>> >>> Thanks, >>> Matyas >>> >> >> >> -- >> Best, >> Hangxiang. >> >
