Lijie Wang created FLINK-30631: ---------------------------------- Summary: Limit the max number of subpartitons consumed by each downstream task Key: FLINK-30631 URL: https://issues.apache.org/jira/browse/FLINK-30631 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Lijie Wang Fix For: 1.17.0
In the current implementation(FLINK-25035), when the upstream vertex parallelism is much greater than the downstream vertex parallelism, it may lead to a large number of channels in the downstream tasks(for example, A -> B, all to all edge, max parallelism is 1000. If parallelism of A is 1000, parallelism of B is decided to be 1, then the only subtask of B will consume 1000 * 1000 subpartitions), resulting in a large overhead for processing channels. In this ticket, we temporarily address this issue by limiting the max number of subpartitons consumed by each downstream task. The ultimate solution should be to support single channel consume multiple subpartitons. -- This message was sent by Atlassian Jira (v8.20.10#820010)