[
https://issues.apache.org/jira/browse/FLINK-34471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819590#comment-17819590
]
Rui Fan commented on FLINK-34471:
---------------------------------
{quote}I realized point to point connections might be a bit tricky when the
parallelism between tasks changes from being equal to being different because
at runtime we then switch to a different partitioner. So worst case we could
run out of network buffers in this scenario.
{quote}
Do you mean the connection type is forward between 2 tasks and then one
parallelism is changed by rescale api, right? I try to run a demo in 1.18.0
just now, the WebUI still show the FORWARD, but the real connection type is
RESCALE.
Based on it, autoscaler can handle it in following logic:
* When the connection type is FORWARD and the parallelism of 2 tasks are same.
We calculate network memory based on FORWARD connection.
* When the connection type is FORWARD and the parallelism of 2 tasks are not
same. We calculate network memory based on RESCALE connection.
Note: this is my temporary idea. I will investigate and document it in detail
next month.
Also, the webui shows a wrong type. IIUC, FLINK-33123 is fixing it. After that,
we can change the calculating logic again. WDYT?
> Tune network memory as part of Autoscaler Memory Tuning
> -------------------------------------------------------
>
> Key: FLINK-34471
> URL: https://issues.apache.org/jira/browse/FLINK-34471
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: Rui Fan
> Assignee: Maximilian Michels
> Priority: Major
>
> Design doc:
> https://docs.google.com/document/d/19HYamwMaYYYOeH3NRbk6l9P-bBLBfgzMYjfGEPWEbeo/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
