Kezhu Wang created FLINK-21817:
----------------------------------
Summary: FLIP-27 Source might break subtask and split assignment
upon rescale
Key: FLINK-21817
URL: https://issues.apache.org/jira/browse/FLINK-21817
Project: Flink
Issue Type: Improvement
Components: Connectors / Common
Affects Versions: 1.13.0
Reporter: Kezhu Wang
On restoring, splits are add back directly to {{SourceReader}} in
{{SourceOperator}}. In no rescaled restoring, bindings between splits and
subtasks are preserved due to no repartition in
{{RoundRobinOperatorStateRepartitioner}}. But in rescaled restoring, these
operator states will be redistributed cross existing subtasks. This might break
possible assignment from {{SourceEnumerator}}.
Given {{KafkaSource}} as an example, the partition to subtask assignment is
decided by {{KafkaSourceEnumerator.getSplitOwner}}. The mappings will break
after rescaling.
I pushed [a test
case|https://github.com/kezhuw/flink/commit/9dc13cd9d7355d468a6ac8aae1b14b3a267581b6#diff-ad6e86c3757199ac3247687a71f9c34ee67b9ac743ae88a9f660950f27bec6eeR238]
using {{KafkaSource}} for evaluation.
I think it requires api addition to solve in generic and configurable way.
Is it a valid issue ? I am not that sure.
cc [~jqin] [~sewen]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
