[
https://issues.apache.org/jira/browse/FLINK-32124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723821#comment-17723821
]
Gyula Fora commented on FLINK-32124:
------------------------------------
[~Zhanghao Chen] can you please confirm that the current behaviour is actually
always partition alignment? How could we align it even better?
> Add option to enable partition alignment for sources
> ----------------------------------------------------
>
> Key: FLINK-32124
> URL: https://issues.apache.org/jira/browse/FLINK-32124
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: Zhanghao Chen
> Priority: Major
>
> Currently, autoscaler did not consider balancing partitions among source
> tasks. In our production env, partition skew has proven to be a severe
> problem for many jobs. Especially in a job topology with all forward or
> rescale shuffles, partition skew on the source side can further lead to data
> imbalance in later operators.
> We should add an option to enable partition alignment for sources for that,
> but making it disabled by default as this has a side effect in that partition
> usu. has limited factors and enabling alignment will greatly limit our
> scaling choices. Also, if data among partitions are imbalanced in the first
> place, partition alignment won't help as well (this is not a common case
> inside our company though).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
