[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613926#comment-17613926
]
Gyula Fora commented on FLINK-29501:
------------------------------------
_--- If the parallelism is being tuned as part of an auto-scaler, wouldn't it
make sense to start from a blank slate if a job is fully resubmitted? I'd
assume the initial parallelism to be a reasonable base-line._
The user may not be aware of the decisions made by the autoscaler, and if the
jobgraph did not structurally change (jobvertexes are the same with same id) I
think it would be a very bad approach to reset the parallelism to the original
one. Of course if we allow this to be configurable we at least do not force one
approach, users can decide to set the config or not.
_---Next you're gonna propose to be able to set the max parallelism, uid, name
etc etc when submitting a job._
That's a bit of a stretch here :) as far as I understand there was no such
intentions there but in any case I don't really see the negative impact. There
is clearly a use-case for this and a configuration based approach is required
for components like the flink-kubernetes-operator. So far we did not hit any
such limitations because almost "everything" is configurable in Flink already.
This is just filling some gaps.
> Allow overriding JobVertex parallelisms during job submission
> -------------------------------------------------------------
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
