Hey Kunal The stack trace you provided for the broker is from a non-controller node. The actual error occurs at the controller while processing the AllocateProduceIds API. Could you please check the controller at that time and let us know what kind of errors you see over there which are correlated with this time frame?
Thanks! Regards, Divij Vaidya On Thu, Jan 12, 2023 at 3:13 PM Kunal Jadhav <kunal.jad...@nciportal.com.invalid> wrote: > Hello All, > > We are facing some issues in the 3.2.3 kafka version cluster. We have > implemented 3 brokers cluster on a single node server in the kubernetes > environment. After every week are facing issues which are listed below. So, > please help us to resolve these issues. Thanks to all in advance. > > *Issues we faced on brokers :* > > [2023-01-12 04:43:52,101] ERROR [KafkaApi-0] Unexpected error handling > request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, > > clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, > correlationId=2) -- InitProducerIdRequestData(transactionalId=null, > transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with > context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, > apiVersion=4, > clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, > correlationId=2), > connectionId='10.233.80.1:9092-10.233.104.6:35952-208286', > clientAddress=/10.233.104.6, principal=User:ANONYMOUS, > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > clientInformation=ClientInformation(softwareName=apache-kafka-java, > softwareVersion=3.2.3), fromPrivilegedListener=true, > > principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3a6b516c > ]) > (kafka.server.KafkaApis) > org.apache.kafka.common.errors.UnknownServerException: The server > experienced an unexpected error when processing the request. > [2023-01-12 04:43:52,159] WARN [RPC ProducerId Manager 0]: Had an > unknown error from the controller, giving up. > (kafka.coordinator.transaction.RPCProducerIdManager) > [2023-01-12 04:43:52,159] ERROR [KafkaApi-0] Unexpected error handling > request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, > > clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, > correlationId=2) -- InitProducerIdRequestData(transactionalId=null, > transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with > context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, > apiVersion=4, > clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, > correlationId=2), > connectionId='10.233.80.1:9092-10.233.104.6:35946-208286', > clientAddress=/10.233.104.6, principal=User:ANONYMOUS, > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > clientInformation=ClientInformation(softwareName=apache-kafka-java, > softwareVersion=3.2.3), fromPrivilegedListener=true, > > principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3042f6ba > ]) > (kafka.server.KafkaApis) > org.apache.kafka.common.errors.UnknownServerException: The server > experienced an unexpected error when processing the request. > [2023-01-12 04:43:52,194] WARN [RPC ProducerId Manager 0]: Had an > unknown error from the controller, giving up. > (kafka.coordinator.transaction.RPCProducerIdManager) > > > *Issues we faced on client :* > > 06:09:22.752 ERROR org.apache.kafka.clients.producer.internals.Sender > - [Producer clientId=javaproducer11675630941604] Aborting producer > batches due to fatal error > org.apache.kafka.common.KafkaException: Unexpected error in > InitProducerIdResponse; The server experienced an unexpected error > when processing the request. > at > org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:1418) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1322) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:418) > [kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316) > [kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) > [kafka-clients-3.2.3.jar:?] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] > > > --- > Thanks & Regards, > Kunal Jadhav >
