Here's the ticket: https://issues.apache.org/jira/browse/KAFKA-12984
And the root cause of that itself: https://issues.apache.org/jira/browse/KAFKA-12983 On Tue, Jun 22, 2021 at 6:15 PM Sophie Blee-Goldman <sop...@confluent.io> wrote: > Hey Tao, > > We recently discovered a bug in the way that the consumer tracks partition > metadata which may cause the cooperative-sticky assignor to throw this > exception in the case of a consumer that dropped out of the group at some > point. I'm just about to file a ticket for it, and it should be fixed in > the upcoming releases. > > The problem is that some consumers are claiming to own partitions that > they no longer actually own after having dropped out. If you can narrow > down the problematic consumers and restart them, it should resolve the > issue. I believe you should be able to tell which consumers are claiming > partitions they no longer own based on the logs, but another option is just > to restart all the consumers (or do a rolling restart until the problem > goes away). > > I'll follow up here with the ticket link once I've filed it. > > -Sophie > > On Tue, Jun 22, 2021 at 12:07 PM Tao Huang <sandy.huang...@gmail.com> > wrote: > >> Thanks for the feedback. >> >> It seems the referred bug is on the server (Broker) side? I just checked >> my Kafka Broker version, it is actually on 2.4.1. So the bug seems does not >> apply to my case. >> >> Should I downgrade my client (Java library) version to 2.4.1? >> >> Thanks! >> >> On 2021/06/21 20:04:31, Ran Lupovich <ranlupov...@gmail.com> wrote: >> > https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-12890 >> > >> > Check out this jira ticket >> > >> > בתאריך יום ב׳, 21 ביוני 2021, 22:15, מאת Tao Huang < >> > sandy.huang...@gmail.com>: >> > >> > > Hi There, >> > > >> > > I am experiencing intermittent issue when consumer group stuck on >> > > "Completing-Reblalance" state. When this is happening, client throws >> error >> > > as below: >> > > >> > > 2021-06-18 13:55:41,086 ERROR io.mylab.adapter.KafkaListener >> > > [edfKafkaListener:CIO.PandC.CIPG.InternalLoggingMetadataInfo] >> Exception on >> > > Kafka listener (InternalLoggingMetadataInfo) - Some partitions are >> > > unassigned but all consumers are at maximum capacity >> > > java.lang.IllegalStateException: Some partitions are unassigned but >> all >> > > consumers are at maximum capacity >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractStickyAssignor.constrainedAssign(AbstractStickyAssignor.java:248) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractStickyAssignor.assign(AbstractStickyAssignor.java:81) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.CooperativeStickyAssignor.assign(CooperativeStickyAssignor.java:64) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractPartitionAssignor.assign(AbstractPartitionAssignor.java:66) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:686) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:111) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:599) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:562) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1151) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1126) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1296) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) >> > > at >> > > >> > > >> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) >> > > at io.mylab.adapter.KafkaListener.run(EdfKafkaListener.java:93) >> > > at java.lang.Thread.run(Thread.java:748) >> > > >> > > The option to exit the state is to stop some members of the consumer >> group. >> > > >> > > Version: 2.6.1 >> > > PARTITION_ASSIGNMENT_STRATEGY: CooperativeStickyAssignor >> > > >> > > Would you please advise what would be the condition to trigger such >> issue? >> > > >> > > Thanks! >> > > >> > >> >
