[jira] [Commented] (FLINK-33090) CheckpointsCleaner clean individual checkpoint states in parallel
[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769854#comment-17769854 ] Yi Zhang commented on FLINK-33090: -- some of our users jobs having large number of state files, this is an annoying issue for them, really appreciate if can resolve this. [@rkhachatryan|https://github.com/rkhachatryan] can you kindly help review as requested? thank you! > CheckpointsCleaner clean individual checkpoint states in parallel > - > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Major > Labels: pull-request-available > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33090) CheckpointsCleaner clean individual checkpoint states in parallel
[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779619#comment-17779619 ] Yi Zhang commented on FLINK-33090: -- [~masteryhx], [~pnowojski] really appreciate your patient and constructive reviews and guidance on working with Flink code, thank you! > CheckpointsCleaner clean individual checkpoint states in parallel > - > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33090) CheckpointsCleaner clean individual checkpoint states in parallel
[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765423#comment-17765423 ] Hangxiang Yu commented on FLINK-33090: -- Thanks for the proposal. We also saw related problem in FLINK-26590 which I just linked. Would you like to contribute your codes ? > CheckpointsCleaner clean individual checkpoint states in parallel > - > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Yi Zhang >Priority: Major > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33090) CheckpointsCleaner clean individual checkpoint states in parallel
[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765458#comment-17765458 ] Yi Zhang commented on FLINK-33090: -- Yes I would like to contribute, I will send a PR for review. > CheckpointsCleaner clean individual checkpoint states in parallel > - > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Yi Zhang >Priority: Major > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33090) CheckpointsCleaner clean individual checkpoint states in parallel
[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765536#comment-17765536 ] Yi Zhang commented on FLINK-33090: -- [~masteryhx] Can you help review? I am new to Flink and greatly appreciate guidance and advices, thank you! > CheckpointsCleaner clean individual checkpoint states in parallel > - > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Major > Labels: pull-request-available > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33090) CheckpointsCleaner clean individual checkpoint states in parallel
[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766182#comment-17766182 ] Hangxiang Yu commented on FLINK-33090: -- Sure. Thanks for the pr. I will take a look asap. > CheckpointsCleaner clean individual checkpoint states in parallel > - > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Major > Labels: pull-request-available > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
