Hi Mathieu, Your understanding is correct, just a few remarks:
1. "Marking the coordinator ... dead for group ..." this is because the heartbeat protocol is actually mutual: i.e. from the consumer's point of view, if it has not received a response, it will mark the current remembered coordinator as "dead" and try to re-discovery the coordinator. Note this is not a WARN / ERROR log since it is actually normal in many cases, since as long as it has re-discovered the coordinator, which still acks itself as the member of the group (i.e. its first HB to the newly founded coordinator, which may just be the old one, returns no error code) then the consumer assumes that it still owns the assigned partitions and hence continue happy fetching. Only when the coordinator has really kicked itself out of the group, then upon re-connecting it may receive an error code in HB response and hence tries to re-join, and if it happens to piggy-back a commit request together you will see the "the group has already rebalanced .." error message. 2. For your case, it is likely that the broker is OK but the client is soft failed, like a long GC. try increasing the "max.poll.interval.ms" and see if it helps (that controls the maximum period a client can live without calling poll() to send another HB). 3. Re: > A deeper look into the specific state of the client, network, and broker at this time interval might suggest an underlying infrastructure issue that should be investigated. Could you share what underlying infrastructure issue are you referring to? BTW there is some related issue for the commit failed exception that has been fixed in trunk post 0.10.2: https://github.com/apache/kafka/commit/4db048d61206bc6efbd143d6293216b7cb4b86c5 Guozhang On Fri, Mar 10, 2017 at 1:01 PM, Mathieu Fenniak < [email protected]> wrote: > Attempting to answer my own questions... but I'd love if someone pointed > out any misunderstandings. :-) > > What causes the Kafka client to "Marking the coordinator ... dead for group > > ..."? Is it untimely heartbeat messages? > > > Small number of causes, but for a running application and a Kafka broker > that didn't die, untimely heartbeats are probably occurring. Maybe an > intermittent network blip, or high load (I/O?) on the network broker. > > Any thoughts on what would cause the commit to fail after a "dead" like > > that, even though it is well within my max.poll.interval.ms? > > > My configuration had default session.timeout.ms (10 seconds) and > heartbeat.interval.ms (3 seconds). If the intermittent network or load > issue disappeared but took 60 seconds, as suggested by my logs, my > consumer's session would have timed out on the server. > > Probably bumping up my session.timeout.ms is the immediate answer (and > possibly heartbeat.interval.ms, just to keep it reasonable). A deeper > look > into the specific state of the client, network, and broker at this time > interval might suggest an underlying infrastructure issue that should be > investigated. > > Is the message "the group has already rebalanced and assigned the > > partitions to another member" really necessarily true? > > > The partitions are revoked from this app's consumer session. If there's no > other consumers, they message's reference to "another member" is a tiny bit > misleading, but the point that they're not assigned to "me" is accurate. > > Mathieu > > > On Fri, Mar 10, 2017 at 11:15 AM, Mathieu Fenniak < > [email protected]> wrote: > > > Hey Kafka Users, > > > > I've been observing a few instances of CommitFailedException (group has > > already rebalanced) that seem to happen well-within max.poll.interval.ms > > since the last commit. In at least one specific case that I've looked > at, > > between the last successful commit and the failed commit, there is a > > "Marking the coordinator ... dead for group ..." message. > > > > Log excerpts / sequence, including consumer config dump: > > https://gist.github.com/mfenniak/46113b4e3cbe35cc54ee103cb0515f34 This > > is a Kafka Streams application, with Kafka Streams & brokers on 0.10.2.0. > > > > What causes the Kafka client to "Marking the coordinator ... dead for > > group ..."? Is it untimely heartbeat messages? > > > > Any thoughts on what would cause the commit to fail after a "dead" like > > that, even though it is well within my max.poll.interval.ms? > > > > Is the message "the group has already rebalanced and assigned the > > partitions to another member" really necessarily true? Because in this > > specific instance, there was only one application instance running in > this > > consumer group, so I'm a little skeptical. > > > > Thanks all, > > > > Mathieu > > > > > -- -- Guozhang
