[ https://issues.apache.org/jira/browse/FLINK-35165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839806#comment-17839806 ]
Venkata krishnan Sowrirajan commented on FLINK-35165: ----------------------------------------------------- JFYI, I am working on the fix for this issue. > AdaptiveBatch Scheduler should not restrict the default source parallelism to > the max parallelism set > ----------------------------------------------------------------------------------------------------- > > Key: FLINK-35165 > URL: https://issues.apache.org/jira/browse/FLINK-35165 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination > Reporter: Venkata krishnan Sowrirajan > Priority: Major > > Copy-pasting the reasoning mentioned on this [discussion > thread|https://lists.apache.org/thread/o887xhvvmn2rg5tyymw348yl2mqt23o7]. > Let me state why I think > "{_}jobmanager.adaptive-batch-scheduler.default-source-parallelism{_}" should > not be bound by the > "{_}jobmanager.adaptive-batch-scheduler.max-parallelism{_}". > * Source vertex is unique and does not have any upstream vertices - > Downstream vertices read shuffled data partitioned by key, which is not the > case for the Source vertex > * Limiting source parallelism by downstream vertices' max parallelism is > incorrect > * If we say for ""semantic consistency" the source vertex parallelism has to > be bound by the overall job's max parallelism, it can lead to following > issues: > ** High filter selectivity with huge amounts of data to read > ** Setting high "*jobmanager.adaptive-batch-scheduler.max-parallelism*" so > that source parallelism can be set higher can lead to small blocks and > sub-optimal performance. > ** Setting high "*jobmanager.adaptive-batch-scheduler.max-parallelism*" > requires careful tuning of network buffer configurations which is unnecessary > in cases where it is not required just so that the source parallelism can be > set high. -- This message was sent by Atlassian Jira (v8.20.10#820010)