Martijn Visser created FLINK-41100:
--------------------------------------
Summary: DynamicKafkaSource shares one split output between
partitions whose split ids collide
Key: FLINK-41100
URL: https://issues.apache.org/jira/browse/FLINK-41100
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: kafka-5.0.0, kafka-4.0.1, kafka-3.2.0
Reporter: Martijn Visser
Assignee: Martijn Visser
{{DynamicKafkaSourceSplit#splitId}} is {{<cluster>-<topic>-<partition>}}, so
cluster {{a}} with topic {{b-c}} and cluster {{a-b}} with topic {{c}} both get
{{a-b-c-0}}. The sub-readers of {{DynamicKafkaSourceReader}} emit into the same
{{ReaderOutput}} under that id ({{KafkaPartitionSplitReaderWrapper}} prefixes
the cluster id), and {{createOutputForSplit}} returns the existing output for
an id it already knows. When one reader holds both partitions, they share one
split-local watermark generator, and when one of them finishes,
{{SourceReaderBase}} releases the output the other one still writes to. From
then on that partition no longer holds back the combined watermark, so its
records can be late. Watermark alignment pauses and resumes both partitions
together for the same reason.
The split ids used towards Flink should be unique per physical partition.
FLINK-41088 fixes the same collision in the enumerator's recovery paths.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
