Zhanghao Chen created FLINK-32872:
-------------------------------------
Summary: Add option to control the default partitioner when the
parallelism of upstream and downstream operator does not match
Key: FLINK-32872
URL: https://issues.apache.org/jira/browse/FLINK-32872
Project: Flink
Issue Type: New Feature
Components: Runtime / Configuration
Affects Versions: 1.17.0
Reporter: Zhanghao Chen
*Problem*
Currently, when the no partitioner is specified, FORWARD partitioner is used if
the parallelism of upstream and downstream operator matches, REBALANCE
partitioner used otherwise. However, this behavior is not configurable and can
be undesirable in certain cases:
# REBALANCE partitioner will create an all-to-all connection between upstream
and downstream operators and consume a lot of extra CPU and memory resources
when the parallelism is high in pipelining mode and RESCALE partitioner is
desirable in this case.
# For Flink SQL jobs, users cannot specify the partitioner directly so far.
And for DataStream jobs, users may not want to explicitly set the partitioner
everywhere.
*Proposal*
Add an option to control the default partitioner when the parallelism of
upstream and downstream operator does not match. The option can have the name
"pipeline.default-partioner-with-unmatched-parallelism" with REBALANCE as the
default value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
