Hi Yun,

Sorry but we didn't understand your questions.
The delay we are experiencing is on the *read* side.
The message is written to kafka topic and consumed by flink with a delay
that depends on the checkpoints interval
When we disabled the checkpoints the messages are immediately consumed
We use the EXACTLY-ONCE semantic.

Please advise.
BR,
Nick

‫בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת ‪Yun Gao‬‏ <‪yungao...@aliyun.com
‬‏>:‬

> Hi nick,
>
>    Sorry I initially think that the data is also write into Kafka with
> flink . So it could be ensured that there is no delay in the write side,
> right ? Does the delay in the read side keeps existing ?
>
> Best,
>  Yun
>
>
>
> ------------------Original Mail ------------------
> *Sender:*nick toker <nick.toker....@gmail.com>
> *Send Date:*Tue Dec 22 01:43:50 2020
> *Recipients:*Yun Gao <yungao...@aliyun.com>
> *CC:*user <user@flink.apache.org>
> *Subject:*Re: checkpoint delay consume message
>
>> hi
>>
>> i am confused
>>
>> the delay in in the source when reading message not on the sink
>>
>> nick
>>
>> ‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪
>> yungao...@aliyun.com‬‏>:‬
>>
>>>  Hi Nick,
>>>
>>>     Are you using EXACTLY_ONCE semantics ? If so the sink would use
>>> transactions, and only commit the transaction on checkpoint complete to
>>> ensure end-to-end exactly-once. A detailed description could be find in [1]
>>>
>>>
>>> Best,
>>>  Yun
>>>
>>>
>>> [1]
>>> https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
>>>
>>> ------------------------------------------------------------------
>>> Sender:nick toker<nick.toker....@gmail.com>
>>> Date:2020/12/21 23:52:34
>>> Recipient:user<user@flink.apache.org>
>>> Theme:checkpoint delay consume message
>>>
>>> Hello,
>>>
>>> We noticed the following behavior:
>>> If we enable the flink checkpoints, we saw that there is a delay between
>>> the time we write a message to the KAFKA topic and the time the flink kafka
>>> connector consumes this message.
>>> The delay is closely related to checkpointInterval and/or
>>> minPauseBetweenCheckpoints meening that the MAX delay when consuming a
>>> message from KAFKA will be one of these parameters.
>>>
>>> Could you please advise how we can remove/control this delay?
>>>
>>> we use flink 1.11.2
>>>
>>> BR
>>> nick
>>>
>>>

Reply via email to