Hi, We had the same problem when running 0.9 consumer against 0.10 Kafka. Upgrading Flink Kafka connector to 0.10 fixed our issue.
Br, Henkka On Mon, Jan 9, 2017 at 5:39 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi, > > Not sure what might be going on here. I’m pretty certain that for > FlinkKafkaConsumer09 when checkpointing is turned off, the internally used > KafkaConsumer client will auto commit offsets back to Kafka at a default > interval of 5000ms (the default value for “auto.commit.interval.ms”). > > Could you perhaps provide the logs of your job (you can send them to me > privately if you prefer to)? > From the logs we should be able to see if the internal KafkaConsumer > client is correctly configured to auto commit and also check if anything > strange is going on. > > Also, how are you reading the committed offsets in Kafka? I recall there > was a problem with the 08 consumer that resulted in the Kafka cli not > correctly showing committed offsets of consumer groups. > However, the 08 consumer had this problem only because we had to implement > the auto offset committing ourselves. I don’t think this should be a issue > for the 09 consumer, since we’re solely relying on the Kafka client’s own > implementation to do the auto offset committing. > > Cheers, > Gordon > > > On January 9, 2017 at 7:55:33 PM, Timo Walther (twal...@apache.org) wrote: > > I'm not a Kafka expert but maybe Gordon (in CC) knows more. > > Timo > > > Am 09/01/17 um 11:51 schrieb Renjie Liu: > > Hi, all: > > I'm using flink 1.1.3 and kafka consumer 09. I read its code and it > > says that the kafka consumer will turn on auto offset commit if > > checkpoint is not enabled. I've turned off checkpoint and it seems > > that kafka client is not committing to offsets to kafka? The offset is > > important for helping us monitoring. Anyone has encountered this before? > > -- > > Liu, Renjie > > Software Engineer, MVAD > > >
