JunRuiLee commented on PR #24736:
URL: https://github.com/apache/flink/pull/24736#issuecomment-2401197362
Sorry for the delayed response due to the National Day holiday in China.
Currently, there are three key fields related to parallelism:
1. defaultMaxParallelism: This comes from the config option values
auto-parallelism.max-parallelism or default.parallelism.
2. globalDefaultSourceParallelism: This comes from config option value
default-source-parallelism or field defaultMaxParallelism.
3. JobVertex max parallelism: This is either user-configured by calling
setMaxParallelism method or fallback to defaultMaxParallelism.
> I think the issue is, if source parallelism is not explicitly set and if
`execution.batch.adaptive.auto-parallelism.max-parallelism` is set <
`execution.batch.adaptive.auto-parallelism.default-source-parallelism` then the
source vertex max parallelism is set to the value of
`execution.batch.adaptive.auto-parallelism.max-parallelism`.
>
The main issue here is that I think if a user has set the max parallelism of
the source (called setMaxParallelism method), we must respect this max
parallelism. However, the current implementation does not account for this and
only considers globalDefaultSourceParallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
