[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765423#comment-17765423 ]
Hangxiang Yu commented on FLINK-33090: -------------------------------------- Thanks for the proposal. We also saw related problem in FLINK-26590 which I just linked. Would you like to contribute your codes ? > CheckpointsCleaner clean individual checkpoint states in parallel > ----------------------------------------------------------------- > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing > Affects Versions: 1.17.1 > Reporter: Yi Zhang > Priority: Major > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)