Hi, Matyas. Do you mean something like adjusting checkpoint intervals dynamically or frequency of uploading files according to the pressure of the durable storage ?
On Wed, Sep 13, 2023 at 9:12 AM Őrhidi Mátyás <matyas.orh...@gmail.com> wrote: > Hey folks, > > Is it possible to add some sort of jitter to the checkpointing logic for > massively parallel jobs to mitigate the burst impact on the durable storage > when a checkpoint is triggered? > > Thanks, > Matyas > -- Best, Hangxiang.
