Attempting to answer my own questions... but I'd love if someone pointed out any misunderstandings. :-)
What causes the Kafka client to "Marking the coordinator ... dead for group > ..."? Is it untimely heartbeat messages? Small number of causes, but for a running application and a Kafka broker that didn't die, untimely heartbeats are probably occurring. Maybe an intermittent network blip, or high load (I/O?) on the network broker. Any thoughts on what would cause the commit to fail after a "dead" like > that, even though it is well within my max.poll.interval.ms? My configuration had default session.timeout.ms (10 seconds) and heartbeat.interval.ms (3 seconds). If the intermittent network or load issue disappeared but took 60 seconds, as suggested by my logs, my consumer's session would have timed out on the server. Probably bumping up my session.timeout.ms is the immediate answer (and possibly heartbeat.interval.ms, just to keep it reasonable). A deeper look into the specific state of the client, network, and broker at this time interval might suggest an underlying infrastructure issue that should be investigated. Is the message "the group has already rebalanced and assigned the > partitions to another member" really necessarily true? The partitions are revoked from this app's consumer session. If there's no other consumers, they message's reference to "another member" is a tiny bit misleading, but the point that they're not assigned to "me" is accurate. Mathieu On Fri, Mar 10, 2017 at 11:15 AM, Mathieu Fenniak < [email protected]> wrote: > Hey Kafka Users, > > I've been observing a few instances of CommitFailedException (group has > already rebalanced) that seem to happen well-within max.poll.interval.ms > since the last commit. In at least one specific case that I've looked at, > between the last successful commit and the failed commit, there is a > "Marking the coordinator ... dead for group ..." message. > > Log excerpts / sequence, including consumer config dump: > https://gist.github.com/mfenniak/46113b4e3cbe35cc54ee103cb0515f34 This > is a Kafka Streams application, with Kafka Streams & brokers on 0.10.2.0. > > What causes the Kafka client to "Marking the coordinator ... dead for > group ..."? Is it untimely heartbeat messages? > > Any thoughts on what would cause the commit to fail after a "dead" like > that, even though it is well within my max.poll.interval.ms? > > Is the message "the group has already rebalanced and assigned the > partitions to another member" really necessarily true? Because in this > specific instance, there was only one application instance running in this > consumer group, so I'm a little skeptical. > > Thanks all, > > Mathieu > >
