Ah, I may have seen this error before. Dhirendra Singh, If you grep your logs, you may find an IllegalStateException or two.
https://issues.apache.org/jira/browse/KAFKA-12948 You need to upgrade to 2.7.2 if this is the issue you're hitting. Kind regards, Liam Clarke-Hutchinson On Sun, 6 Mar 2022 at 04:30, Mailbox - Dhirendra Kumar Singh < dhirendr...@gmail.com> wrote: > Let me rephrase my issue. > Issue occur when broker loose connectivity to zookeeper server. > Connectivity loss can happen due to many reasons…zookeeper servers getting > bounced, due to some network glitch etc… > > After the brokers reconnect to zookeeper server I expect the kafka cluster > to come back in stable state by itself without any manual intervention. > but instead few partitions remain under replicated due to the error I > pasted earlier. > > I feel this is some kind of bug. I am going to file a bug. > > > > Thanks, > > Dhirendra. > > > > From: Thomas Cooper <c...@tomcooper.dev> > Sent: Friday, March 4, 2022 7:01 PM > To: Dhirendra Singh <dhirendr...@gmail.com> > Cc: users@kafka.apache.org > Subject: Re: Few partitions stuck in under replication > > > > Do you roll the controller last? > > I suspect this is more to do with the way you are rolling the cluster > (which I am still not clear on the need for) rather than some kind of bug > in Kafka (though that could of course be the case). > > Tom > > On 04/03/2022 01:59, Dhirendra Singh wrote: > > Hi Tom, > During the rolling restart we check for under replicated partition count > to be zero in the readiness probe before restarting the next POD in order. > This issue never occurred before. It started after we upgraded kafka > version from 2.5.0 to 2.7.1. > So i suspect some bug introduced in the version after 2.5.0. > > > > Thanks, > > Dhirendra. > > > > On Thu, Mar 3, 2022 at 11:09 PM Thomas Cooper <c...@tomcooper.dev <mailto: > c...@tomcooper.dev> > wrote: > > I suspect this nightly rolling will have something to do with your issues. > If you are just rolling the stateful set in order, with no dependence on > maintaining minISR and other Kafka considerations you are going to hit > issues. > > If you are running on Kubernetes I would suggest using an Operator like > Strimzi <https://strimzi.io/> which will do a lot of the Kafka admin > tasks like this for you automatically. > > Tom > > On 03/03/2022 16:28, Dhirendra Singh wrote: > > Hi Tom, > > Doing the nightly restart is the decision of the cluster admin. I have no > control on it. > We have implementation using stateful set. restart is triggered by > updating a annotation in the pod. > Issue is not triggered by kafka cluster restart but the zookeeper servers > restart. > > Thanks, > > Dhirendra. > > > > On Thu, Mar 3, 2022 at 7:19 PM Thomas Cooper <c...@tomcooper.dev <mailto: > c...@tomcooper.dev> > wrote: > > Hi Dhirenda, > > Firstly, I am interested in why are you restarting the ZK and Kafka > cluster every night? > > Secondly, how are you doing the restarts. For example, in [Strimzi]( > https://strimzi.io/), when we roll the Kafka cluster we leave the > designated controller broker until last. For each of the other brokers we > wait until all the partitions they are leaders for are above their minISR > and then we roll the broker. In this way we maintain availability and make > sure leadership can move off the rolling broker temporarily. > > Cheers, > > Tom Cooper > > [@tomncooper](https://twitter.com/tomncooper) | https://tomcooper.dev > > On 03/03/2022 07:38, Dhirendra Singh wrote: > > > Hi All, > > > > We have kafka cluster running in kubernetes. kafka version we are using > is > > 2.7.1. > > Every night zookeeper servers and kafka brokers are restarted. > > After the nightly restart of the zookeeper servers some partitions remain > > stuck in under replication. This happens randomly but not at every > nightly > > restart. > > Partitions remain under replicated until kafka broker with the partition > > leader is restarted. > > For example partition 4 of consumer_offsets topic remain under replicated > > and we see following error in the log... > > > > [2022-02-28 04:01:20,217] WARN [Partition __consumer_offsets-4 broker=1] > > Controller failed to update ISR to PendingExpandIsr(isr=Set(1), > > newInSyncReplicaId=2) due to unexpected UNKNOWN_SERVER_ERROR. Retrying. > > (kafka.cluster.Partition) > > [2022-02-28 04:01:20,217] ERROR [broker-1-to-controller] Uncaught error > in > > request completion: (org.apache.kafka.clients.NetworkClient) > > java.lang.IllegalStateException: Failed to enqueue `AlterIsr` request > with > > state LeaderAndIsr(leader=1, leaderEpoch=2728, isr=List(1, 2), > > zkVersion=4719) for partition __consumer_offsets-4 > > at kafka.cluster.Partition.sendAlterIsrRequest(Partition.scala:1403) > > at > > > kafka.cluster.Partition.$anonfun$handleAlterIsrResponse$1(Partition.scala:1438) > > at kafka.cluster.Partition.handleAlterIsrResponse(Partition.scala:1417) > > at > > > kafka.cluster.Partition.$anonfun$sendAlterIsrRequest$1(Partition.scala:1398) > > at > > > kafka.cluster.Partition.$anonfun$sendAlterIsrRequest$1$adapted(Partition.scala:1398) > > at > > > kafka.server.AlterIsrManagerImpl.$anonfun$handleAlterIsrResponse$8(AlterIsrManager.scala:166) > > at > > > kafka.server.AlterIsrManagerImpl.$anonfun$handleAlterIsrResponse$8$adapted(AlterIsrManager.scala:163) > > at scala.collection.immutable.List.foreach(List.scala:333) > > at > > > kafka.server.AlterIsrManagerImpl.handleAlterIsrResponse(AlterIsrManager.scala:163) > > at > > > kafka.server.AlterIsrManagerImpl.responseHandler$1(AlterIsrManager.scala:94) > > at > > > kafka.server.AlterIsrManagerImpl.$anonfun$sendRequest$2(AlterIsrManager.scala:104) > > at > > > kafka.server.BrokerToControllerRequestThread.handleResponse(BrokerToControllerChannelManagerImpl.scala:175) > > at > > > kafka.server.BrokerToControllerRequestThread.$anonfun$generateRequests$1(BrokerToControllerChannelManagerImpl.scala:158) > > at > > > org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) > > at > > > org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:586) > > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:578) > > at > kafka.common.InterBrokerSendThread.doWork(InterBrokerSendThread.scala:71) > > at > > > kafka.server.BrokerToControllerRequestThread.doWork(BrokerToControllerChannelManagerImpl.scala:183) > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) > > Looks like some kind of race condition bug...anyone has any idea ? > > > > Thanks, > > Dhirendra > > > > -- > > Tom Cooper > > @tomncooper <https://twitter.com/tomncooper> | tomcooper.dev < > https://tomcooper.dev> > > > > -- > > Tom Cooper > > @tomncooper <https://twitter.com/tomncooper> | tomcooper.dev < > https://tomcooper.dev> > >
