Hey Kamesh, thank you for the question. Could you also check the broker side log to see if the group is forming generations properly? Information we have for now is a bit hard to tell what's going on. Also since you have upgraded to 2.3, during incremental rebalancing you will experience 2 rebalance in a row but won't revoke/assign tasks unless necessary, could you verify that for old connectors their partitions are not getting revoked during the first rebalance?
Boyang On Sat, Aug 3, 2019 at 1:40 AM Kamesh <kam.iit...@gmail.com> wrote: > Hi, > I am using Kafka connect cluster for writing data to S3. I have observed, > whenever I add a new connector or update the config of an existing > connector, I think group balancing is happening and it is affecting all the > existing connectors. Rebalancing is happening for all of the existing > connectors also. All of my log files are filled with the following log > messages > > *[2019-08-03 08:28:28,668] INFO [Consumer clientId=connector-consumer-xxx, > groupId=connect-xxx] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)* > > This rebalancing is taking a very long time, and sometimes not evening > completing. Any pointers on this? > > I am using Kafka connect *2.3.0, *and this version supports incremental > rebalancing and it should not affect existing connectors. Am I > missing something? > > I have already tuned *session.timeout.ms <http://session.timeout.ms>* > and *max.poll.interval.ms > <http://max.poll.interval.ms>* configs and increased their values as > suggested in the community. > > > Thanks & Regards > Kamesh. >
