Hi, Nick ~ The behavior is as expected, because Kafka source/sink relies on the Checkpoints to complement the exactly-once write semantics, a checkpoint snapshot the states on a time point which is used for recovering, the current internals for Kafka sink is that it writes to Kafka but only commits it when a checkpoint completes.
For your needs, i guess you want a more near-real-time write but still keep the exactly once semantics, i'm sorry to tell that there is no other infrastructure that we can use for exactly-once semantics except for the checkpoints. nick toker <nick.toker....@gmail.com> 于2020年12月27日周日 下午3:12写道： > Hi > > any idea? > is it a bug? > > > regards' > nick > > בתאריך יום ד׳, 23 בדצמ׳ 2020 ב-11:10 מאת nick toker < > nick.toker....@gmail.com>: > >> Hello >> >> We noticed the following behavior: >> If we enable the flink checkpoints, we saw that there is a delay between >> the time we write a message to the KAFKA topic and the time the flink kafka >> connector consumes this message. >> The delay is closely related to checkpointInterval and/or >> minPauseBetweenCheckpoints meening that the MAX delay when consuming a >> message from KAFKA will be one of these parameters >> >> If we disable the checkpoints, the message is immediately consumed >> We work with the EXACTLY_ONCE semantic >> Please note that we inject only one message >> >> Could you please advise how we can remove/control this delay? >> >> Please see the attached code of AbstractFetcher and KafkaFetcher (as a >> png file) >> (For example emitRecordsWithTimestamps() use a lock on checkpointLock). >> Could this explain the behaviour ? >> >> >> BR >> >
