[
https://issues.apache.org/jira/browse/KAFKA-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471522#comment-16471522
]
qiuyueqiong commented on KAFKA-5611:
------------------------------------
after applied https://github.com/apache/kafka/pull/3571 on version 0.10.2.0, we
encountered an infinite loop occurs in ConsumerCoordinator.java:
java.lang.IllegalStateException: Coordinator selected invalid assignment
protocol: null
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:208)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:305)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
for some reason, the ConsumerCoordinator's resetGeneration() is called, and the
protocol of generation is set to null.
then in method joinGroupIfNeeded() of
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.java
,onJoinComplete() method throws an IllegalStateException,
and the resetJoinGroupFuture() method will never has the chance to be call:
if (future.succeeded()) {
onJoinComplete(generation.generationId, generation.memberId,
generation.protocol, future.value());
// We reset the join group future only after the completion
callback returns. This ensures
// that if the callback is woken up, we will retry it on the
next joinGroupIfNeeded.
resetJoinGroupFuture();
needsJoinPrepare = true;
the joinFuture is not clear to null, JoinGroupRequest will not be sent in the
next loop
is this an issue?
> One or more consumers in a consumer-group stop consuming after rebalancing
> --------------------------------------------------------------------------
>
> Key: KAFKA-5611
> URL: https://issues.apache.org/jira/browse/KAFKA-5611
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.10.2.0
> Reporter: Panos Skianis
> Assignee: Jason Gustafson
> Priority: Major
> Labels: reliability
> Fix For: 0.11.0.1
>
> Attachments: Server 1, Server 2, Server 3,
> bad-server-with-more-logging-1.tar.gz, kka02
>
>
> Scenario:
> - 3 zookeepers, 4 Kafkas. 0.10.2.0, with 0.9.0 compatibility still on
> (other apps need it but the one mentioned below is already on kafka 0.10.2.0
> client).
> - 3 servers running 1 consumer each under the same consumer groupId.
> - Servers seem to be consuming messages happily but then there is a timeout
> to an external service that causes our app to restart the Kafka Consumer on
> one of the servers (this is by design). That causes rebalancing of the group
> and upon restart of one of the Consumers seem to "block".
> - Server 3 is where the problems occur.
> - Problem fixes itself either by restarting one of the 3 servers or cause
> the group to rebalance again by using the console consumer with the
> autocommit set to false and using the same group.
>
> Note:
> - Haven't managed to recreate it at will yet.
> - Mainly happens in production environment, often enough. Hence I do not
> have any logs with DEBUG/TRACE statements yet.
> - Extracts from log of each app server are attached. Also the log of the
> kafka that seems to be dealing with the related group and generations.
> - See COMMENT lines in the files for further info.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
