[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[
https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325601#comment-17325601
]
Till Rohrmann commented on FLINK-21844:
---
Yes, the idea would be to load the savepoint and then extract from it potential
constraints for the {{ExecutionGraph}} creation [~nicholasjiang].
> Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
>
>
> Key: FLINK-21844
> URL: https://issues.apache.org/jira/browse/FLINK-21844
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Konstantin Knauf
>Assignee: Austin Cawley-Edwards
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> I believe we should not automatically change the maxParallelism when the
> "scheduler-mode" is set to "reactive", because:
> * it magically breaks savepoint compatibility, when you switch between
> default and reactive scheduler mode
> * the maximum parallelism is an orthogonal concern that in my opinion should
> not be mixed with the scheduler mode. The reactive scheduler should respect
> the maxParallelism, but it should not set/ change its default value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[ https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323765#comment-17323765 ] Nicholas Jiang commented on FLINK-21844: [~trohrmann], do you mean that the restore from savepoint shoud be before the ExecutionGraph creation? Now, loading the savepoint and validating the ExecutionJobVertex is after the creation of ExecutionGraph. > Do not auto-configure maxParallelism when setting "scheduler-mode: reactive" > > > Key: FLINK-21844 > URL: https://issues.apache.org/jira/browse/FLINK-21844 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Konstantin Knauf >Assignee: Austin Cawley-Edwards >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > I believe we should not automatically change the maxParallelism when the > "scheduler-mode" is set to "reactive", because: > * it magically breaks savepoint compatibility, when you switch between > default and reactive scheduler mode > * the maximum parallelism is an orthogonal concern that in my opinion should > not be mixed with the scheduler mode. The reactive scheduler should respect > the maxParallelism, but it should not set/ change its default value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[ https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319357#comment-17319357 ] Chesnay Schepler commented on FLINK-21844: -- I've created FLINK-22234 as the follow-up. > Do not auto-configure maxParallelism when setting "scheduler-mode: reactive" > > > Key: FLINK-21844 > URL: https://issues.apache.org/jira/browse/FLINK-21844 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Konstantin Knauf >Assignee: Austin Cawley-Edwards >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > I believe we should not automatically change the maxParallelism when the > "scheduler-mode" is set to "reactive", because: > * it magically breaks savepoint compatibility, when you switch between > default and reactive scheduler mode > * the maximum parallelism is an orthogonal concern that in my opinion should > not be mixed with the scheduler mode. The reactive scheduler should respect > the maxParallelism, but it should not set/ change its default value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[
https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319257#comment-17319257
]
Till Rohrmann commented on FLINK-21844:
---
[~austince], [~chesnay] do we already have a follow up ticket for fixing this
problem properly (moving reading the savepoint out of the {{ExecutionGraph}}
creation)? If not, please create on and link it here.
> Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
>
>
> Key: FLINK-21844
> URL: https://issues.apache.org/jira/browse/FLINK-21844
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Konstantin Knauf
>Assignee: Austin Cawley-Edwards
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> I believe we should not automatically change the maxParallelism when the
> "scheduler-mode" is set to "reactive", because:
> * it magically breaks savepoint compatibility, when you switch between
> default and reactive scheduler mode
> * the maximum parallelism is an orthogonal concern that in my opinion should
> not be mixed with the scheduler mode. The reactive scheduler should respect
> the maxParallelism, but it should not set/ change its default value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[
https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306047#comment-17306047
]
Till Rohrmann commented on FLINK-21844:
---
I think if the user does not configure an explicit {{maxParallelism}}, then
Flink should fall back to the default setting as defined in
https://github.com/apache/flink/blob/37a8e3cac73587fb845617c6accd1c51def306e6/flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyGroupRangeAssignment.java#L137.
But we shouldn't fail since we did not fail before.
> Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
>
>
> Key: FLINK-21844
> URL: https://issues.apache.org/jira/browse/FLINK-21844
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Konstantin Knauf
>Assignee: Austin Cawley-Edwards
>Priority: Major
> Fix For: 1.13.0
>
>
> I believe we should not automatically change the maxParallelism when the
> "scheduler-mode" is set to "reactive", because:
> * it magically breaks savepoint compatibility, when you switch between
> default and reactive scheduler mode
> * the maximum parallelism is an orthogonal concern that in my opinion should
> not be mixed with the scheduler mode. The reactive scheduler should respect
> the maxParallelism, but it should not set/ change its default value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[ https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305213#comment-17305213 ] Austin Cawley-Edwards commented on FLINK-21844: --- Does this mean a user must configure `maxParallelism` for all operators in reactive mode, otherwise Flink will produce an exception when building the `ExecutionGraph`? > Do not auto-configure maxParallelism when setting "scheduler-mode: reactive" > > > Key: FLINK-21844 > URL: https://issues.apache.org/jira/browse/FLINK-21844 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Konstantin Knauf >Assignee: Austin Cawley-Edwards >Priority: Major > Fix For: 1.13.0 > > > I believe we should not automatically change the maxParallelism when the > "scheduler-mode" is set to "reactive", because: > * it magically breaks savepoint compatibility, when you switch between > default and reactive scheduler mode > * the maximum parallelism is an orthogonal concern that in my opinion should > not be mixed with the scheduler mode. The reactive scheduler should respect > the maxParallelism, but it should not set/ change its default value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[ https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304148#comment-17304148 ] Till Rohrmann commented on FLINK-21844: --- Sounds good to me. > Do not auto-configure maxParallelism when setting "scheduler-mode: reactive" > > > Key: FLINK-21844 > URL: https://issues.apache.org/jira/browse/FLINK-21844 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Konstantin Knauf >Priority: Major > > I believe we should not automatically change the maxParallelism when the > "scheduler-mode" is set to "reactive", because: > * it magically breaks savepoint compatibility, when you switch between > default and reactive scheduler mode > * the maximum parallelism is an orthogonal concern that in my opinion should > not be mixed with the scheduler mode. The reactive scheduler should respect > the maxParallelism, but it should not set change its default value. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[ https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304107#comment-17304107 ] Konstantin Knauf commented on FLINK-21844: -- I agree with Chesnay. If the user wants to run with a higher parallelism (w/ or w/o adaptive scheduler), she should/needs to increase the maxParallelism. This is also in line with the expectation that the name "maxParallelism" raises. > Do not auto-configure maxParallelism when setting "scheduler-mode: reactive" > > > Key: FLINK-21844 > URL: https://issues.apache.org/jira/browse/FLINK-21844 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Konstantin Knauf >Priority: Major > > I believe we should not automatically change the maxParallelism when the > "scheduler-mode" is set to "reactive", because: > * it magically breaks savepoint compatibility, when you switch between > default and reactive scheduler mode > * the maximum parallelism is an orthogonal concern that in my opinion should > not be mixed with the scheduler mode. The reactive scheduler should respect > the maxParallelism, but it should not set change its default value. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[ https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304094#comment-17304094 ] Chesnay Schepler commented on FLINK-21844: -- There's been a recent discussion in FLINK-21695 about raising the default maxParallelism and handling the associated migration/compatibility/performance concerns. I think we should just stick to the current defaults (128) for now and document it accordingly that users may want to increase this. > Do not auto-configure maxParallelism when setting "scheduler-mode: reactive" > > > Key: FLINK-21844 > URL: https://issues.apache.org/jira/browse/FLINK-21844 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Konstantin Knauf >Priority: Major > > I believe we should not automatically change the maxParallelism when the > "scheduler-mode" is set to "reactive", because: > * it magically breaks savepoint compatibility, when you switch between > default and reactive scheduler mode > * the maximum parallelism is an orthogonal concern that in my opinion should > not be mixed with the scheduler mode. The reactive scheduler should respect > the maxParallelism, but it should not set change its default value. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21844) Do not auto-configure maxParallelism when setting "scheduler-mode: reactive"
[ https://issues.apache.org/jira/browse/FLINK-21844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304061#comment-17304061 ] Till Rohrmann commented on FLINK-21844: --- I see the problem and I agree that this is confusing for our users. Looking at the code, it seems that we set the max parallelism to 128 if neither parallelism nor max parallelism has been configured. I am wondering whether this value would be too low for some users who will try out the reactive mode. > Do not auto-configure maxParallelism when setting "scheduler-mode: reactive" > > > Key: FLINK-21844 > URL: https://issues.apache.org/jira/browse/FLINK-21844 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Konstantin Knauf >Priority: Major > > I believe we should not automatically change the maxParallelism when the > "scheduler-mode" is set to "reactive", because: > * it magically breaks savepoint compatibility, when you switch between > default and reactive scheduler mode > * the maximum parallelism is an orthogonal concern that in my opinion should > not be mixed with the scheduler mode. The reactive scheduler should respect > the maxParallelism, but it should not set change its default value. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
