Hi, > I read this as, "The offsets committed to Kafka are ignored, the offsets committed within a checkpoint are used".
yes, exactly > So from the sounds of things, regardless of the consumer group's offsets, it will always start from a checkpoint or savepoints offsets if there are some (unless checkpointing offsets is turned off). Yes. But, keep in mind this part: > setStartFromGroupOffsets (default behaviour): Start reading partitions from the consumer group’s (group.id setting in the consumer properties) committed offsets in Kafka brokers.* If offsets could not be found for a partition, the auto.offset.reset setting in the properties will be used.* As I understand it, if you are using the default `setStartFromGroupOffsets`, and you happen to change `group.id` (which is what I believe you were asking about in the first e-mail), after changing the `group.id` FlinkKafkaConsumer will not be able to found previously saved offsets in the Flink's state and it will start reading from completely new set of offsets. The same way as if this would be a freshly started new job without any state. Those new offsets would be as specified/defined via `auto.offset.reset`. Piotrek pon., 18 sty 2021 o 18:12 Rex Fenley <r...@remind101.com> napisał(a): > Thank you, > > Some parts that stick out > >The Flink Kafka Consumer allows configuring the behaviour of how offsets > are committed back to Kafka brokers. Note that the Flink Kafka Consumer > does not rely on the committed offsets for fault tolerance guarantees. The > committed offsets are only a means to expose the consumer’s progress for > monitoring purposes. > > I read this as, "The offsets committed to Kafka are ignored, the offsets > committed within a checkpoint are used". > > >With Flink’s checkpointing enabled, the Flink Kafka Consumer will consume > records from a topic and periodically checkpoint all its Kafka offsets, > together with the state of other operations. In case of a job failure, > Flink will restore the streaming program to the state of the latest > checkpoint and re-consume the records from Kafka, starting from the offsets > that were stored in the checkpoint. > > This seems to say something similar. > > So from the sounds of things, regardless of the consumer group's offsets, > it will always start from a checkpoint or savepoints offsets if there are > some (unless checkpointing offsets is turned off). > > Is this interpretation correct? > > Thanks! > > > On Mon, Jan 18, 2021 at 3:23 AM Piotr Nowojski <pnowoj...@apache.org> > wrote: > >> Hi Rex, >> >> I believe this section answers your question [1] >> >> Piotrek >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-start-position-configuration >> >> pon., 18 sty 2021 o 09:00 赵一旦 <hinobl...@gmail.com> napisał(a): >> >>> If you changed the consumer group in your new job, the group id will be >>> the new one you set. >>> The job will continue to consumer the topics from the >>> savepoint/checkpoint you specified no matter whether the group id is the >>> original one? >>> >>> Rex Fenley <r...@remind101.com> 于2021年1月18日周一 下午12:53写道： >>> >>>> Hello, >>>> >>>> When using the Kafka consumer connector, if we restore a from a >>>> checkpoint or savepoint using a differently named consumer group than the >>>> one we originally ran a job with will it still pick up exactly where it >>>> left off or are you locked into using the same consumer group as before? >>>> >>>> Thanks! >>>> >>>> -- >>>> >>>> Rex Fenley | Software Engineer - Mobile and Backend >>>> >>>> >>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >>>> <https://www.facebook.com/remindhq> >>>> >>> > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | > FOLLOW US <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> >
