Hi nick,
Sorry I initially think that the data is also write into Kafka with flink .
So it could be ensured that there is no delay in the write side, right ? Does
the delay in the read side keeps existing ?
Best,
Yun
------------------Original Mail ------------------
Sender:nick toker <nick.toker....@gmail.com>
Send Date:Tue Dec 22 01:43:50 2020
Recipients:Yun Gao <yungao...@aliyun.com>
CC:user <user@flink.apache.org>
Subject:Re: checkpoint delay consume message
hi
i am confused
the delay in in the source when reading message not on the sink
nick
בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת Yun Gao <yungao...@aliyun.com>:
Hi Nick,
Are you using EXACTLY_ONCE semantics ? If so the sink would use
transactions, and only commit the transaction on checkpoint complete to ensure
end-to-end exactly-once. A detailed description could be find in [1]
Best,
Yun
[1]
https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
------------------------------------------------------------------
Sender:nick toker<nick.toker....@gmail.com>
Date:2020/12/21 23:52:34
Recipient:user<user@flink.apache.org>
Theme:checkpoint delay consume message
Hello,
We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the
time we write a message to the KAFKA topic and the time the flink kafka
connector consumes this message.
The delay is closely related to checkpointInterval and/or
minPauseBetweenCheckpoints meening that the MAX delay when consuming a message
from KAFKA will be one of these parameters.
Could you please advise how we can remove/control this delay?
we use flink 1.11.2
BR
nick
