Hi suman,
FlinkKafkaConsumer.setStartFromLatest() means you always consume messages
from the latest whenever you restart the flink job，the consumer ignore any
committed group offsets.
auto.offset.reset=latest means the consumer fetch messages from the latest
if you never committed any offsets before, it is a default strategy if the
consumer does not found any offsets from brokers.
At 2021-08-05 06:35:05, "suman shil" <cncf.s...@gmail.com> wrote:
In my flink streaming application I have kafka datasource. I am using the kafka
property auto.offset.reset=latest. I am wondering if I need to use
FlinkKafkaConsumer.setStartFromLatest(). Are they similar? Can I use either of
them? Following is the documentation from flink code.
/**
* Specifies the consumer to start reading from the latest offset for all
partitions. This lets
* the consumer ignore any committed group offsets in Zookeeper / Kafka brokers.
*
* <p>This method does not affect where partitions are read from when the
consumer is restored
* from a checkpoint or savepoint. When the consumer is restored from a
checkpoint or savepoint,
* only the offsets in the restored state will be used.
*
* @return The consumer object, to allow function chaining.
*/
public FlinkKafkaConsumerBase<T> setStartFromLatest() {
this.startupMode = StartupMode.LATEST;
this.startupOffsetsTimestamp = null;
this.specificStartupOffsets = null;
return this;
}
Thanks
