[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-26590: --- Labels: auto-deprioritized-major pull-request-available stale-assigned (was: pull-request-available stale-assigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.14.3, 1.15.0 >Reporter: Roman Khachatryan >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-26590: --- Labels: pull-request-available stale-assigned stale-major (was: pull-request-available stale-assigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.14.3, 1.15.0 >Reporter: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned, stale-major > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-26590: -- Fix Version/s: (was: 1.17.0) > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.14.3, 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-26590: -- Issue Type: Improvement (was: Bug) > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.14.3, 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.16.0 > > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-26590: -- Fix Version/s: 1.17.0 (was: 1.16.0) > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.14.3, 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.17.0 > > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingbo Huang updated FLINK-26590: - Issue Type: Bug (was: Improvement) > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.3, 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.16.0 > > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-26590: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.15.0, 1.14.3 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.16.0 > > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.1#820001)