Re: Is group.id required in Kafka connector for offsets to be stored in checkpoint?

2019-02-18 Thread sohimankotia
Yes Konstantin Knauf-2 . You are right . -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Is group.id required in Kafka connector for offsets to be stored in checkpoint?

2019-02-18 Thread Konstantin Knauf
Hi David, Hi Sohi, this should not be the case. If a savepoint/checkpoint is provided, Flink should always take the offsets from the state regardless of the `group.id` provided. Which Flink version and which FlinkKafkaConsumer version do you use? Best, Konstantin On Mon, Feb 18, 2019 at 5:50

Re: Is group.id required in Kafka connector for offsets to be stored in checkpoint?

2019-02-17 Thread sohimankotia
Hi David, We are also running streaming jobs over Kafka source . Yes : Consumer Group Id needs to be set for Kafka source explicitly t . We are also using checkpointing and save points for persisting state . Any time we change group id it starts from latest offset(default Kafka connector

Is group.id required in Kafka connector for offsets to be stored in checkpoint?

2019-02-16 Thread David Corley
We've got a relatively simply job that reads in from Kafka, and writes to S3. We've had a couple of job failures where the consumer lag had built up, but after the restart, the lag was wiped out because our offset positions were lost and we consumed from latest offset. The job has checkpointing