There seems to be a double-post with the mail "Long latency when consuming
a message from KAFKA and checkpoint is enabled". Let's continue discussion
there.

On Sun, Dec 27, 2020 at 8:33 AM nick toker <nick.toker....@gmail.com> wrote:

> Hi,
> Hi,  We think  we are using the default values unless we are missing
> something.
> So this doesn't explain the problem we are facing.
> Could you please tell us how to choose synchronous or asynchronous
> checkpoints just to be sure we are using the correct configuration ?
> BR,
> Nick
>
> ‫בתאריך יום ה׳, 24 בדצמ׳ 2020 ב-3:36 מאת ‪lec ssmi‬‏ <‪
> shicheng31...@gmail.com‬‏>:‬
>
>> Checkpoint can be done synchronously and  asynchronously,  the latter is
>> the default .
>> If you chooese  the synchronous way , it may cause this problem.
>>
>> nick toker <nick.toker....@gmail.com> 于2020年12月23日周三 下午3:53写道:
>>
>>> Hi Yun,
>>>
>>> Sorry but we didn't understand your questions.
>>> The delay we are experiencing is on the *read* side.
>>> The message is written to kafka topic and consumed by flink with a delay
>>> that depends on the checkpoints interval
>>> When we disabled the checkpoints the messages are immediately consumed
>>> We use the EXACTLY-ONCE semantic.
>>>
>>> Please advise.
>>> BR,
>>> Nick
>>>
>>> ‫בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת ‪Yun Gao‬‏ <‪
>>> yungao...@aliyun.com‬‏>:‬
>>>
>>>> Hi nick,
>>>>
>>>>    Sorry I initially think that the data is also write into Kafka with
>>>> flink . So it could be ensured that there is no delay in the write side,
>>>> right ? Does the delay in the read side keeps existing ?
>>>>
>>>> Best,
>>>>  Yun
>>>>
>>>>
>>>>
>>>> ------------------Original Mail ------------------
>>>> *Sender:*nick toker <nick.toker....@gmail.com>
>>>> *Send Date:*Tue Dec 22 01:43:50 2020
>>>> *Recipients:*Yun Gao <yungao...@aliyun.com>
>>>> *CC:*user <user@flink.apache.org>
>>>> *Subject:*Re: checkpoint delay consume message
>>>>
>>>>> hi
>>>>>
>>>>> i am confused
>>>>>
>>>>> the delay in in the source when reading message not on the sink
>>>>>
>>>>> nick
>>>>>
>>>>> ‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪
>>>>> yungao...@aliyun.com‬‏>:‬
>>>>>
>>>>>>  Hi Nick,
>>>>>>
>>>>>>     Are you using EXACTLY_ONCE semantics ? If so the sink would use
>>>>>> transactions, and only commit the transaction on checkpoint complete to
>>>>>> ensure end-to-end exactly-once. A detailed description could be find in 
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>>  Yun
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>> https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
>>>>>>
>>>>>> ------------------------------------------------------------------
>>>>>> Sender:nick toker<nick.toker....@gmail.com>
>>>>>> Date:2020/12/21 23:52:34
>>>>>> Recipient:user<user@flink.apache.org>
>>>>>> Theme:checkpoint delay consume message
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We noticed the following behavior:
>>>>>> If we enable the flink checkpoints, we saw that there is a delay
>>>>>> between the time we write a message to the KAFKA topic and the time the
>>>>>> flink kafka connector consumes this message.
>>>>>> The delay is closely related to checkpointInterval and/or
>>>>>> minPauseBetweenCheckpoints meening that the MAX delay when consuming a
>>>>>> message from KAFKA will be one of these parameters.
>>>>>>
>>>>>> Could you please advise how we can remove/control this delay?
>>>>>>
>>>>>> we use flink 1.11.2
>>>>>>
>>>>>> BR
>>>>>> nick
>>>>>>
>>>>>>

-- 

Arvid Heise | Senior Java Developer

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng

Reply via email to