[ https://issues.apache.org/jira/browse/KAFKA-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471522#comment-16471522 ]
qiuyueqiong commented on KAFKA-5611: ------------------------------------ after applied https://github.com/apache/kafka/pull/3571 on version 0.10.2.0, we encountered an infinite loop occurs in ConsumerCoordinator.java: java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:208) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:305) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286) for some reason, the ConsumerCoordinator's resetGeneration() is called, and the protocol of generation is set to null. then in method joinGroupIfNeeded() of org.apache.kafka.clients.consumer.internals.AbstractCoordinator.java ,onJoinComplete() method throws an IllegalStateException, and the resetJoinGroupFuture() method will never has the chance to be call: if (future.succeeded()) { onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value()); // We reset the join group future only after the completion callback returns. This ensures // that if the callback is woken up, we will retry it on the next joinGroupIfNeeded. resetJoinGroupFuture(); needsJoinPrepare = true; the joinFuture is not clear to null, JoinGroupRequest will not be sent in the next loop is this an issue? > One or more consumers in a consumer-group stop consuming after rebalancing > -------------------------------------------------------------------------- > > Key: KAFKA-5611 > URL: https://issues.apache.org/jira/browse/KAFKA-5611 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.10.2.0 > Reporter: Panos Skianis > Assignee: Jason Gustafson > Priority: Major > Labels: reliability > Fix For: 0.11.0.1 > > Attachments: Server 1, Server 2, Server 3, > bad-server-with-more-logging-1.tar.gz, kka02 > > > Scenario: > - 3 zookeepers, 4 Kafkas. 0.10.2.0, with 0.9.0 compatibility still on > (other apps need it but the one mentioned below is already on kafka 0.10.2.0 > client). > - 3 servers running 1 consumer each under the same consumer groupId. > - Servers seem to be consuming messages happily but then there is a timeout > to an external service that causes our app to restart the Kafka Consumer on > one of the servers (this is by design). That causes rebalancing of the group > and upon restart of one of the Consumers seem to "block". > - Server 3 is where the problems occur. > - Problem fixes itself either by restarting one of the 3 servers or cause > the group to rebalance again by using the console consumer with the > autocommit set to false and using the same group. > > Note: > - Haven't managed to recreate it at will yet. > - Mainly happens in production environment, often enough. Hence I do not > have any logs with DEBUG/TRACE statements yet. > - Extracts from log of each app server are attached. Also the log of the > kafka that seems to be dealing with the related group and generations. > - See COMMENT lines in the files for further info. -- This message was sent by Atlassian JIRA (v7.6.3#76005)