[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614078#comment-17614078
]
Maximilian Michels commented on FLINK-29501:
--------------------------------------------
Let me try to summarize:
It is a valid use case for external systems to observe the job via task level
metrics. Based on these observations, we may want to automatically alter the
job vertex parallelisms. There is no programmatic way to do this with Flink now
if the k8s application mode is used where we don't have direct control over the
JobGraph. Also, the observed job can have **any** topology and we cannot assume
the entry point respects parallelism overrides as arguments in the main method.
There are two possible solutions here:
# Provide the overrides as part of the configuration
# Issue a call to the rescale API
The problem with (2) is that the Recale API is both disabled and broken at the
moment, see https://issues.apache.org/jira/browse/FLINK-12312 It has
fundamental issues like not being able to use the HA mode with it. I'm afraid,
this makes option (1) the only viable option to me because, frankly, I'm not
sure the issues of the rescale mode can easily be fixed.
> Allow overriding JobVertex parallelisms during job submission
> -------------------------------------------------------------
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
