Hi all, thanks for the KIP. The motivation for addressing this improvement in this shape (client-side metrics) is not too clear to me, mainly given how the rebalance landscape has changed drastically (KIP-848 GA).
I do get that the way to troubleshoot rebalances with the classic consumer is troublesome (based on client logs), visibility limited etc, but the problem is actually much bigger than that, and that's why the new Consumer protocol (KIP-848) was delivered and changed the whole experience around rebalances (including visibility). Then, with KIP-848, rebalances are driven by the coordinator, so I would expect that if there are visibility gaps we should better address them on the broker side? (e.g, more metrics if needed?). Also there is a clear timeline to make the new Consumer protocol default for the KafkaConsumer and eventually deprecate the classic consumer (KIP-1274), so we should keep that in mind, just to better assess/target the investment. Was this considered? Thoughts? Thanks! Lianet On Mon, Jul 20, 2026 at 9:40 AM Alieh Saeedi via dev <[email protected]> wrote: > Hi Aditya, > > Thanks for the KIP. It certainly addresses a long-standing gap. Do you > please cite KAFKA-12472: https://issues.apache.org/jira/browse/KAFKA-12472 > in the KIP? and also explain what has changed to make the taxonomy > tractable now when it wasn't in 2021 — given that the intervening change > (KIP-848) arguably cuts the other way. > > -Alieh > > On Thu, Jul 16, 2026 at 7:01 PM Andrew Schofield <[email protected]> > wrote: > > > Hi Aditya, > > Thanks for the KIP. > > > > I can see the idea behind the KIP but I'm not convinced that it's > possible > > to work out a coherent set of RebalanceCause cases, in particular for the > > consumer group protocol. In this protocol, rebalances are always > > "coordinator-initiated". Just because the application calls > > Consumer.subscribe() resulting in a ConsumerGroupHeartbeat request being > > sent to the GC, it's not certain that the next ConsumerGroupHeartbeat > > response will contain the newly assigned partitions. Assignment is > > asynchronous and the GC makes various optimisations to avoid excessive > > rebalances for groups with a lot of churn. I'm not sure I could tell the > > difference between "coordinator-initiated" and "subscription-changed". If > > you can explain in more detail how each of the cases corresponds to > > ConsumerGroupHeartbeat response, it would help. > > > > We do need to put more work into the broker-side metrics for rebalance > for > > the consumer group protocol. We'd like more visibility than we have today > > for sure. > > > > Thanks, > > Andrew > > > > On 2026/07/08 21:19:20 Aditya Kousik wrote: > > > Hi all, > > > > > > I’d like to start a discussion on KIP-1364 as a follow-up to KIP-1306. > > > > > > KIP link: > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/jYE_Gg__;!!Ayb5sqE7!vtE4w2WCvGgNm36QYBGcUm1inI112efwGiCTLfM8P2zNLenxQkZ0W_d1olR0JKB9Umy0Na1blvL4a7Az7PhqCMc$ > > > > > > A Kafka consumer client today cannot tell why a rebalance happened > > without grepping logs. The rebalance trigger can help with triaging: > > > > > > * max.poll.interval.ms exceeded implies misconfig or delayed poll > loop; > > > * Member fencing points at session timeout > > > * Coordinator-initiated rebalances are regular occurrences due to join > > and leave requests. > > > * Application calls to Consumer.enforceRebalance()are expected and the > > application owns the context. > > > > > > Existing rebalance metrics describe frequency and latency but carry no > > cause dimension: > > > > > > rebalance-rate-per-hour, rebalance-latency-{avg,max}, rebalance-total > > > last-rebalance-seconds-ago, last-poll-seconds-ago > > > > > > So, the cause would be useful in diagnosing issues by surfacing the > > event to the user. > > > > > > Looking forward to hearing your thoughts on this. > > > > > > Best, > > > Aditya Kousik > > >
