[ https://issues.apache.org/jira/browse/FLINK-26018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494672#comment-17494672 ]
Piotr Nowojski commented on FLINK-26018: ---------------------------------------- Yes [~danderson], you are right. Thanks for spotting the typo. > Unnecessary late events when using the new KafkaSource > ------------------------------------------------------ > > Key: FLINK-26018 > URL: https://issues.apache.org/jira/browse/FLINK-26018 > Project: Flink > Issue Type: Bug > Components: Connectors / Common > Affects Versions: 1.14.3 > Reporter: Jun Qin > Assignee: Qingsheng Ren > Priority: Critical > Labels: pull-request-available > Fix For: 1.15.0, 1.14.4 > > Attachments: message in kafka.txt, > taskmanager_10.28.0.131_33249-b3370c_log > > > There is an issue with the new KafkaSource connector in Flink 1.14: when one > task consumes messages from multiple topic partitions (statically created, > timestamp are in order), it may start with one partition and advances > watermarks before the data from other partitions come. In this case, the > early messages in other partitions may unnecessarily be considered as late > ones. > I discussed with [~renqs], it seems that the new KafkaSource only adds a > partition into {{WatermarkMultiplexer}} when it receives data from that > partition. In contrast, FlinkKafkaConsumer adds all known partition before it > fetch any data. > Attached two files: the messages in Kafka and the corresponding TM logs. -- This message was sent by Atlassian Jira (v8.20.1#820001)