Re: Difference between different values for starting offset

2023-07-04 Thread Hang Ruan
Hi, Oscar. Yes, your are right. If starting from a checkpoint or savepoint, kafka connector will always use the offset stored in the states. If starting without a checkpoint or savepoint, kafka connector will use the specific startup mode. Best, Hang Oscar Perez via user 于2023年7月4日周二 20:54写道:

Re: Difference between different values for starting offset

2023-07-04 Thread Oscar Perez via user
Hei, Ok, thanks. so if I understand this correctly the difference between OffsetInitializer.earliest and commitedOffset(OffsetResetStrategy.EARLIEST) will be in the case that there is no flink state. In this case, earliest will not check kafka committed offset and start from earliest while in the l

Re: Difference between different values for starting offset

2023-07-03 Thread Mason Chen
Hi Oscar, You are correct about the OffsetInitializer being only effective when there is no Flink state--in addition, if you have partition discovery on, this initializer will be reused for the new partitions (i.e. splits) discovered. Assuming the job is continuing from the offset in Flink state,

Difference between different values for starting offset

2023-07-03 Thread Oscar Perez via user
Hei, Looking at the flink documentation for kafkasource I see the following values for starting offset: OffsetInitializer.earliest OffsetInitializer.latest OffsetInitializer.commitedOffset(OffsetResetStrategy.EARLIEST) >From what I understand OffsetInitializer.earliest uses earliest offset the f