[
https://issues.apache.org/jira/browse/FLINK-41100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-41100:
-----------------------------------
Fix Version/s: kafka-5.1.0
> DynamicKafkaSource shares one split output between partitions whose split ids
> collide
> -------------------------------------------------------------------------------------
>
> Key: FLINK-41100
> URL: https://issues.apache.org/jira/browse/FLINK-41100
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: kafka-3.2.0, kafka-4.0.1, kafka-5.0.0
> Reporter: Martijn Visser
> Assignee: Martijn Visser
> Priority: Major
> Fix For: kafka-5.1.0
>
>
> {{DynamicKafkaSourceSplit#splitId}} is {{<cluster>-<topic>-<partition>}}, so
> cluster {{a}} with topic {{b-c}} and cluster {{a-b}} with topic {{c}} both
> get {{a-b-c-0}}. The sub-readers of {{DynamicKafkaSourceReader}} emit into
> the same {{ReaderOutput}} under that id ({{KafkaPartitionSplitReaderWrapper}}
> prefixes the cluster id), and {{createOutputForSplit}} returns the existing
> output for an id it already knows. When one reader holds both partitions,
> they share one split-local watermark generator, and when one of them
> finishes, {{SourceReaderBase}} releases the output the other one still writes
> to. From then on that partition no longer holds back the combined watermark,
> so its records can be late. Watermark alignment pauses and resumes both
> partitions together for the same reason.
> The split ids used towards Flink should be unique per physical partition.
> FLINK-41088 fixes the same collision in the enumerator's recovery paths.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
