I will try to answer. rebalancing triggers when one or two consuemrs(client) leaves the group because of any reason. The thumb rule is Number of partitions should be equal to number of consumer threads. If there are 300 partitions assigned one thread each it wont rebalance untill some consumer marked as dead. How it marks as dead: if the kafka doesnt receive heartbeat from consumer in 5 mins(or defined at client side).
If 20 consumers are dead from kafka persepctive at time T1 then it will trigger rebalance. It will trigger rebalance at time T2 when there is new consumer added to the group and there is poll request from new consumer. If there is no issue with number of partitions and number of consumers then it wont trigger rebalance. Terms I used may not be accurate😊 Regards, Sunil. On Thu, 8 Jun 2023 at 3:59 PM, Edvard Fagerholm <edvard.fagerh...@gmail.com> wrote: > Hello, > > I couldn't find an answer in the documentation to the following. If a new > machine joins a consumer group and Kafka triggers a rebalance, will it > randomly reassign partitions or will it hand over partitions from existing > consumers to the newly joined one? In other words, will it attempt to move > as few partitions as possible between consumers? > > The main implications of this is local in-memory caches and scaling up the > number of machines in a consumer group, since the scaling up operation will > require nuking any local caches for the partitions that were moved. This > would cause a spike on any DBs that are being cached on the consumers. > > Best, > Edvard >
