Re: kafka offset not working

2019-10-08 Thread Benjamin Cuthbert
So looks like checkpoints are not working, each time a job is stopped and started it is a different checkpoint. Using savepoints manually seems to work if you do a call to save the savepoint and then run it with -s savepoint location. Why does checkpoints not work? On Mon, Oct 7, 2019 at 10:25 PM

Re: kafka offset not working

2019-10-07 Thread Eduardo Winpenny Tejedor
Flink only uses the specified starting point consumer-side setting the first time the application is deployed. After that Flink will store the Kafka offsets as part of the (source) operator state and when restarted from a checkpoint/savepoint it'll continue to consume from where it left off. Hope

kafka offset not working

2019-10-07 Thread Benjamin Cuthbert
All Our kafka consumer has the group.id property but when we receive messages on the channel when not connect and then reconnect we don't get the message. The kafka console consumer when running "-group testben" works perfectly fine. Is there some other place to setup flink to read from the