[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612956#comment-17612956
]
Maximilian Michels commented on FLINK-29501:
--------------------------------------------
What you suggest generally works and I've seen this in many user pipelines.
However, I think the intention here is slightly different to the normal
parameterization of Flink applications.
The intention of this JIRA is to allow an external component, e.g. autoscaler,
to change the parallelism of a Flink deployment without changing the logical
topology of the DAG. We can't easily apply parallelism changes based on
observed task metrics on the operator level because, from the perspective of an
external system, tasks are mostly opaque. The goal is to work on the final
JobGraph and prevent any topology changes (e.g. due to chaining, slot sharing,
etc) before the actual JobGraph creation. Further, we may not have control over
the main method of the user at all.
> Allow overriding JobVertex parallelisms during job submission
> -------------------------------------------------------------
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Minor
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
