[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613764#comment-17613764
]
Chesnay Schepler commented on FLINK-29501:
------------------------------------------
?? The upside may be shorter downtime.??
The upside is significantly more potential for optimizations down-the-line,
like deferring a rescaling to a job failure (== low cost rescaling) or it being
properly integrated into batch jobs (==rescale on next stage, similar to the
AdaptiveBatchScheduler).
??rescaling with an api call is not a "persistent" operation. If a user later
wants to upgrade the job using a savepoint, they would have to then tweak the
parallelism of the entire pipeline from the main method or rescale again after
upgrade.??
If the parallelism is being tuned as part of an auto-scaler, wouldn't it make
sense to start from a blank slate if a job is fully resubmitted? I'd assume the
initial parallelism to be a reasonable base-line.
??Flink already provides a myriad of config options including default
parallelism, max parallelism and various execution settings, this is not
different from that in my view.??
This is quite different to me since it introduces the concept of
vertex-specific configurations.
It introduces _yet another_ way how the parallelism can be _configured_.
Next you're gonna propose to be able to set the max parallelism, uid, name etc
etc when submitting a job.
> Allow overriding JobVertex parallelisms during job submission
> -------------------------------------------------------------
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
