Hi, Hi, We think we are using the default values unless we are missing something. So this doesn't explain the problem we are facing. Could you please tell us how to choose synchronous or asynchronous checkpoints just to be sure we are using the correct configuration ? BR, Nick
בתאריך יום ה׳, 24 בדצמ׳ 2020 ב-3:36 מאת lec ssmi < shicheng31...@gmail.com>: > Checkpoint can be done synchronously and asynchronously, the latter is > the default . > If you chooese the synchronous way , it may cause this problem. > > nick toker <nick.toker....@gmail.com> 于2020年12月23日周三 下午3:53写道： > >> Hi Yun, >> >> Sorry but we didn't understand your questions. >> The delay we are experiencing is on the *read* side. >> The message is written to kafka topic and consumed by flink with a delay >> that depends on the checkpoints interval >> When we disabled the checkpoints the messages are immediately consumed >> We use the EXACTLY-ONCE semantic. >> >> Please advise. >> BR, >> Nick >> >> בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת Yun Gao < >> yungao...@aliyun.com>: >> >>> Hi nick, >>> >>> Sorry I initially think that the data is also write into Kafka with >>> flink . So it could be ensured that there is no delay in the write side, >>> right ? Does the delay in the read side keeps existing ? >>> >>> Best, >>> Yun >>> >>> >>> >>> ------------------Original Mail ------------------ >>> *Sender:*nick toker <nick.toker....@gmail.com> >>> *Send Date:*Tue Dec 22 01:43:50 2020 >>> *Recipients:*Yun Gao <yungao...@aliyun.com> >>> *CC:*user <user@flink.apache.org> >>> *Subject:*Re: checkpoint delay consume message >>> >>>> hi >>>> >>>> i am confused >>>> >>>> the delay in in the source when reading message not on the sink >>>> >>>> nick >>>> >>>> בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת Yun Gao < >>>> yungao...@aliyun.com>: >>>> >>>>> Hi Nick, >>>>> >>>>> Are you using EXACTLY_ONCE semantics ? If so the sink would use >>>>> transactions, and only commit the transaction on checkpoint complete to >>>>> ensure end-to-end exactly-once. A detailed description could be find in >>>>> [1] >>>>> >>>>> >>>>> Best, >>>>> Yun >>>>> >>>>> >>>>> [1] >>>>> https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html >>>>> >>>>> ------------------------------------------------------------------ >>>>> Sender:nick toker<nick.toker....@gmail.com> >>>>> Date:2020/12/21 23:52:34 >>>>> Recipient:user<user@flink.apache.org> >>>>> Theme:checkpoint delay consume message >>>>> >>>>> Hello, >>>>> >>>>> We noticed the following behavior: >>>>> If we enable the flink checkpoints, we saw that there is a delay >>>>> between the time we write a message to the KAFKA topic and the time the >>>>> flink kafka connector consumes this message. >>>>> The delay is closely related to checkpointInterval and/or >>>>> minPauseBetweenCheckpoints meening that the MAX delay when consuming a >>>>> message from KAFKA will be one of these parameters. >>>>> >>>>> Could you please advise how we can remove/control this delay? >>>>> >>>>> we use flink 1.11.2 >>>>> >>>>> BR >>>>> nick >>>>> >>>>>
