[jira] [Updated] (FLINK-23381) Provide backpressure (currently job fails if a limit is hit)
[ https://issues.apache.org/jira/browse/FLINK-23381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-23381: --- Labels: pull-request-available (was: ) > Provide backpressure (currently job fails if a limit is hit) > > > Key: FLINK-23381 > URL: https://issues.apache.org/jira/browse/FLINK-23381 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > With the current approach, job will fail if dstl.dfs.upload.max-in-flight > (bytes) is reached. > > Unsetting the limit roughly matches the current behaviour for other backends: > async phase doesn't backpressure > (state.backend.rocksdb.checkpoint.transfer.thread.num only sets the upload > thread pool size which uses an unbounded queue). > > Note that blocking caller in DfsWriter.persistInternal() will also block > regular stream processing (because of pre-emptive writes). This may or may > not be desired behaviour. > > Blocking sync phase of a snapshot can also have some issues (e.g. not being > able to abort the checkpoint) which should be considered. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23381) Provide backpressure (currently job fails if a limit is hit)
[ https://issues.apache.org/jira/browse/FLINK-23381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-23381: -- Fix Version/s: (was: 1.14.0) 1.15.0 > Provide backpressure (currently job fails if a limit is hit) > > > Key: FLINK-23381 > URL: https://issues.apache.org/jira/browse/FLINK-23381 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Priority: Major > Fix For: 1.15.0 > > > With the current approach, job will fail if dstl.dfs.upload.max-in-flight > (bytes) is reached. > > Unsetting the limit roughly matches the current behaviour for other backends: > async phase doesn't backpressure > (state.backend.rocksdb.checkpoint.transfer.thread.num only sets the upload > thread pool size which uses an unbounded queue). > > Note that blocking caller in DfsWriter.persistInternal() will also block > regular stream processing (because of pre-emptive writes). This may or may > not be desired behaviour. > > Blocking sync phase of a snapshot can also have some issues (e.g. not being > able to abort the checkpoint) which should be considered. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23381) Provide backpressure (currently job fails if a limit is hit)
[ https://issues.apache.org/jira/browse/FLINK-23381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-23381: -- Description: With the current approach, job will fail if dstl.dfs.upload.max-in-flight (bytes) is reached. Unsetting the limit roughly matches the current behaviour for other backends: async phase doesn't backpressure (state.backend.rocksdb.checkpoint.transfer.thread.num only sets the upload thread pool size which uses an unbounded queue). Note that blocking caller in DfsWriter.persistInternal() will also block regular stream processing (because of pre-emptive writes). This may or may not be desired behaviour. Blocking sync phase of a snapshot can also have some issues (e.g. not being able to abort the checkpoint) which should be considered. was:With the current approach, job will fail if dstl.dfs.upload.max-in-flight (bytes) is reached. > Provide backpressure (currently job fails if a limit is hit) > > > Key: FLINK-23381 > URL: https://issues.apache.org/jira/browse/FLINK-23381 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Priority: Major > Fix For: 1.14.0 > > > With the current approach, job will fail if dstl.dfs.upload.max-in-flight > (bytes) is reached. > > Unsetting the limit roughly matches the current behaviour for other backends: > async phase doesn't backpressure > (state.backend.rocksdb.checkpoint.transfer.thread.num only sets the upload > thread pool size which uses an unbounded queue). > > Note that blocking caller in DfsWriter.persistInternal() will also block > regular stream processing (because of pre-emptive writes). This may or may > not be desired behaviour. > > Blocking sync phase of a snapshot can also have some issues (e.g. not being > able to abort the checkpoint) which should be considered. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23381) Provide backpressure (currently job fails if a limit is hit)
[ https://issues.apache.org/jira/browse/FLINK-23381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-23381: -- Summary: Provide backpressure (currently job fails if a limit is hit) (was: Provide backpressure (currently job fails if a limit hit)) > Provide backpressure (currently job fails if a limit is hit) > > > Key: FLINK-23381 > URL: https://issues.apache.org/jira/browse/FLINK-23381 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Priority: Major > Fix For: 1.14.0 > > > With the current approach, job will fail if dstl.dfs.upload.max-in-flight > (bytes) is reached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
