Hi Alexey, looking at KafkaTopicPartitionStatus, it looks that it does not contain this information. In a nutshell, what you probably have to do is to aggregate the watermarks across all partitions and then pause the consumption of a partition if its watermark advances too much wrt to the minimum watermark. However, this will stall the whole reading process if there is a partition which has no more data. Hence, you will probably also need a mechanism to advance the watermark if the partition becomes idle.
Note that the community is currently working on a new KafkaConnector based on Flink's new source API (FLIP-27). If I am not mistaken, then these new interfaces should eventually also support event time alignment. Cheers, Till On Fri, May 28, 2021 at 7:17 PM Alexey Trenikhun <yen...@msn.com> wrote: > Hello, > I'm thinking about implementing custom Kafka connector which provides > event alignment (similar to FLINK-10921, which seems abandoned). What is > the way to determine is partition is idle from override > of AbstractFetcher.emitRecordsWithTimestamps()? > Does KafkaTopicPartitionState has this information ? > > Thanks, > Alexey >
