Flink kafka consumers don't honor group.id

2018-06-28 Thread Giriraj
It seems that Flink kafka consumer don't honor group.id while consumers are added dynamically. Lets say I have some flink kafka consumers reading from a topic and I dynamically add some new Flink kafka consumers with same group.id, kafka messages are getting duplicated to existing as well as new co

Re: Flink kafka consumers don't honor group.id

2018-06-28 Thread Tzu-Li (Gordon) Tai
Hi Giriraj, The fact that the Flink Kafka Consumer doesn't use the group.id property, is an expected behavior. Since the partition-to-subtask assignment of the Flink Kafka Consumer needs to be deterministic in Flink, the consumer uses static assignment instead of the more high-level consumer group

Re: Flink kafka consumers don't honor group.id

2018-07-03 Thread Giriraj
Hi Gordon, Gordon:If I understood you correctly, what you are doing is, while a job with a Kafka consumer is already running, you want to start a new job also with a Kafka consumer as the source and uses the same group.id so that the topic's messages are routed between the two jobs. Is this corre