[ https://issues.apache.org/jira/browse/FLINK-32124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723801#comment-17723801 ]
Zhanghao Chen commented on FLINK-32124: --------------------------------------- Thanks [~gyfora]. I'll follow up there. > Add option to enable partition alignment for sources > ---------------------------------------------------- > > Key: FLINK-32124 > URL: https://issues.apache.org/jira/browse/FLINK-32124 > Project: Flink > Issue Type: Improvement > Components: Autoscaler > Reporter: Zhanghao Chen > Priority: Major > > Currently, autoscaler did not consider balancing partitions among source > tasks. In our production env, partition skew has proven to be a severe > problem for many jobs. Especially in a job topology with all forward or > rescale shuffles, partition skew on the source side can further lead to data > imbalance in later operators. > We should add an option to enable partition alignment for sources for that, > but making it disabled by default as this has a side effect in that partition > usu. has limited factors and enabling alignment will greatly limit our > scaling choices. Also, if data among partitions are imbalanced in the first > place, partition alignment won't help as well (this is not a common case > inside our company though). -- This message was sent by Atlassian Jira (v8.20.10#820010)