Zhanghao Chen created FLINK-32124:
-------------------------------------
Summary: Add option to enable partition alignment for sources
Key: FLINK-32124
URL: https://issues.apache.org/jira/browse/FLINK-32124
Project: Flink
Issue Type: Improvement
Components: Autoscaler
Reporter: Zhanghao Chen
Currently, autoscaler did not consider balancing partitions among source tasks.
In our production env, partition skew has proven to be a severe problem for
many jobs. Especially in a job topology with all forward or rescale shuffles,
partition skew on the source side can further lead to data imbalance in later
operators.
We should add an option to enable partition alignment for sources for that, but
making it disabled by default as this has a side effect in that partition usu.
has limited factors and enabling alignment will greatly limit our scaling
choices. Also, if data among partitions are imbalanced in the first place,
partition alignment won't help as well (this is not a common case inside our
company though).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
