[ https://issues.apache.org/jira/browse/FLINK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761944#comment-17761944 ]
Zhanghao Chen commented on FLINK-32872: --------------------------------------- [~huweihua] Looking forward to your suggestions on this issue~ > Add option to control the default partitioner when the parallelism of > upstream and downstream operator does not match > --------------------------------------------------------------------------------------------------------------------- > > Key: FLINK-32872 > URL: https://issues.apache.org/jira/browse/FLINK-32872 > Project: Flink > Issue Type: New Feature > Components: Runtime / Configuration > Affects Versions: 1.17.0 > Reporter: Zhanghao Chen > Priority: Major > > *Problem* > Currently, when the no partitioner is specified, FORWARD partitioner is used > if the parallelism of upstream and downstream operator matches, REBALANCE > partitioner used otherwise. However, this behavior is not configurable and > can be undesirable in certain cases: > # REBALANCE partitioner will create an all-to-all connection between > upstream and downstream operators and consume a lot of extra CPU and memory > resources when the parallelism is high in pipelining mode and RESCALE > partitioner is desirable in this case. > # For Flink SQL jobs, users cannot specify the partitioner directly so far. > And for DataStream jobs, users may not want to explicitly set the partitioner > everywhere. > *Proposal* > Add an option to control the default partitioner when the parallelism of > upstream and downstream operator does not match. The option can have the name > "pipeline.default-partitioner-with-unmatched-parallelism" with REBALANCE as > the default value. -- This message was sent by Atlassian Jira (v8.20.10#820010)