I tried, seems the offset will not be committed when doing savepoint After submitting a flink job, I just use the following cmd to see the committed offset bin/kafka-consumer-groups.sh --bootstrap-server xxx:9092 --describe --group groupName TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG dwd_audio_record 4 8338888 8341364 2476 dwd_audio_record 3 39699731 39702207 2476 dwd_audio_record 2 40014838 40017314 2476 dwd_audio_record 5 8350221 8352698 2477 dwd_audio_record 0 8389365 8391842 2477 dwd_audio_record 1 8342001 8344477 2476
After the flink job does a checkpoint, I can see the CURRENT-OFFSET changed. But when i stop the job with savepoint, the CURRENT-OFFSET not change Thanks, Lei On Thu, Jun 6, 2024 at 7:25 PM Zhanghao Chen <zhanghao.c...@outlook.com> wrote: > Yes, the exact offset position will also be committed when doing the > savepoint. > > Best, > Zhanghao Chen > ------------------------------ > *From:* Lei Wang <leiwang...@gmail.com> > *Sent:* Thursday, June 6, 2024 16:54 > *To:* Zhanghao Chen <zhanghao.c...@outlook.com>; ruanhang1...@gmail.com < > ruanhang1...@gmail.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Force to commit kafka offset when stop a job. > > Thanks Zhanghao && Hang. > > I am familiar with the flink savepoint feature. The exact offset position > is stored in savepoint and the job can be resumed from the savepoint using > the offset position that is stored in it. > But I am not sure whether the exact offset position is committed to kafka > when doing savepoint. If the answer is yes, i can ensure no repeat > consumption even the state is incompatible because i can stop with > savepoint and submit the job without savepoint > Thanks, > Lei > > On Thu, Jun 6, 2024 at 10:50 AM Zhanghao Chen <zhanghao.c...@outlook.com> > wrote: > > Hi, you could stop the job with a final savepoint [1]. Flink which will > trigger a final offset commit on the final savepoint. > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint > > Best, > Zhanghao Chen > ------------------------------ > *From:* Lei Wang <leiwang...@gmail.com> > *Sent:* Wednesday, June 5, 2024 22:47 > *To:* user <user@flink.apache.org> > *Subject:* Force to commit kafka offset when stop a job. > > > When stopping a flink job that consuming kafka message, how to force it to > commit kafka offset > > Thanks, > Lei > >
