Re: Kafka issues related 3.2.3 version zookeeper less cluster
Ok, thanks Divij, I will check it. --- Thanks & Regards, Kunal Jadhav On Mon, Jan 16, 2023 at 7:47 PM Divij Vaidya wrote: > Hey Kunal > > The stack trace you provided for the broker is from a non-controller node. > The actual error occurs at the controller while processing the > AllocateProduceIds API. Could you please check the controller at that time > and let us know what kind of errors you see over there which are correlated > with this time frame? > > Thanks! > > Regards, > Divij Vaidya > > > > On Thu, Jan 12, 2023 at 3:13 PM Kunal Jadhav > wrote: > > > Hello All, > > > > We are facing some issues in the 3.2.3 kafka version cluster. We have > > implemented 3 brokers cluster on a single node server in the kubernetes > > environment. After every week are facing issues which are listed below. > So, > > please help us to resolve these issues. Thanks to all in advance. > > > > *Issues we faced on brokers :* > > > > [2023-01-12 04:43:52,101] ERROR [KafkaApi-0] Unexpected error handling > > request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, > > > > > clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, > > correlationId=2) -- InitProducerIdRequestData(transactionalId=null, > > transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with > > context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, > > apiVersion=4, > > > clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, > > correlationId=2), > > connectionId='10.233.80.1:9092-10.233.104.6:35952-208286', > > clientAddress=/10.233.104.6, principal=User:ANONYMOUS, > > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > > clientInformation=ClientInformation(softwareName=apache-kafka-java, > > softwareVersion=3.2.3), fromPrivilegedListener=true, > > > > > principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3a6b516c > > ]) > > (kafka.server.KafkaApis) > > org.apache.kafka.common.errors.UnknownServerException: The server > > experienced an unexpected error when processing the request. > > [2023-01-12 04:43:52,159] WARN [RPC ProducerId Manager 0]: Had an > > unknown error from the controller, giving up. > > (kafka.coordinator.transaction.RPCProducerIdManager) > > [2023-01-12 04:43:52,159] ERROR [KafkaApi-0] Unexpected error handling > > request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, > > > > > clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, > > correlationId=2) -- InitProducerIdRequestData(transactionalId=null, > > transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with > > context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, > > apiVersion=4, > > > clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, > > correlationId=2), > > connectionId='10.233.80.1:9092-10.233.104.6:35946-208286', > > clientAddress=/10.233.104.6, principal=User:ANONYMOUS, > > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > > clientInformation=ClientInformation(softwareName=apache-kafka-java, > > softwareVersion=3.2.3), fromPrivilegedListener=true, > > > > > principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3042f6ba > > ]) > > (kafka.server.KafkaApis) > > org.apache.kafka.common.errors.UnknownServerException: The server > > experienced an unexpected error when processing the request. > > [2023-01-12 04:43:52,194] WARN [RPC ProducerId Manager 0]: Had an > > unknown error from the controller, giving up. > > (kafka.coordinator.transaction.RPCProducerIdManager) > > > > > > *Issues we faced on client :* > > > > 06:09:22.752 ERROR org.apache.kafka.clients.producer.internals.Sender > > - [Producer clientId=javaproducer11675630941604] Aborting producer > > batches due to fatal error > > org.apache.kafka.common.KafkaException: Unexpected error in > > InitProducerIdResponse; The server experienced an unexpected error > > when processing the request. > > at > > > org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:1418) > > ~[kafka-clients-3.2.3.jar:?] > > at > > > org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1322) > > ~[kafka-clients-3.2.3.jar:?] > > at > > > org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) > > ~[kafka-clients-3.2.3.jar:?] > > at > > > org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583) > > ~[kafka-clients-3.2.3.jar:?] > > at > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575) > > ~[kafka-clients-3.2.3.jar:?] > > at > > >
Re: Kafka issues related 3.2.3 version zookeeper less cluster
Hey Kunal The stack trace you provided for the broker is from a non-controller node. The actual error occurs at the controller while processing the AllocateProduceIds API. Could you please check the controller at that time and let us know what kind of errors you see over there which are correlated with this time frame? Thanks! Regards, Divij Vaidya On Thu, Jan 12, 2023 at 3:13 PM Kunal Jadhav wrote: > Hello All, > > We are facing some issues in the 3.2.3 kafka version cluster. We have > implemented 3 brokers cluster on a single node server in the kubernetes > environment. After every week are facing issues which are listed below. So, > please help us to resolve these issues. Thanks to all in advance. > > *Issues we faced on brokers :* > > [2023-01-12 04:43:52,101] ERROR [KafkaApi-0] Unexpected error handling > request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, > > clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, > correlationId=2) -- InitProducerIdRequestData(transactionalId=null, > transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with > context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, > apiVersion=4, > clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, > correlationId=2), > connectionId='10.233.80.1:9092-10.233.104.6:35952-208286', > clientAddress=/10.233.104.6, principal=User:ANONYMOUS, > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > clientInformation=ClientInformation(softwareName=apache-kafka-java, > softwareVersion=3.2.3), fromPrivilegedListener=true, > > principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3a6b516c > ]) > (kafka.server.KafkaApis) > org.apache.kafka.common.errors.UnknownServerException: The server > experienced an unexpected error when processing the request. > [2023-01-12 04:43:52,159] WARN [RPC ProducerId Manager 0]: Had an > unknown error from the controller, giving up. > (kafka.coordinator.transaction.RPCProducerIdManager) > [2023-01-12 04:43:52,159] ERROR [KafkaApi-0] Unexpected error handling > request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, > > clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, > correlationId=2) -- InitProducerIdRequestData(transactionalId=null, > transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with > context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, > apiVersion=4, > clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, > correlationId=2), > connectionId='10.233.80.1:9092-10.233.104.6:35946-208286', > clientAddress=/10.233.104.6, principal=User:ANONYMOUS, > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > clientInformation=ClientInformation(softwareName=apache-kafka-java, > softwareVersion=3.2.3), fromPrivilegedListener=true, > > principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3042f6ba > ]) > (kafka.server.KafkaApis) > org.apache.kafka.common.errors.UnknownServerException: The server > experienced an unexpected error when processing the request. > [2023-01-12 04:43:52,194] WARN [RPC ProducerId Manager 0]: Had an > unknown error from the controller, giving up. > (kafka.coordinator.transaction.RPCProducerIdManager) > > > *Issues we faced on client :* > > 06:09:22.752 ERROR org.apache.kafka.clients.producer.internals.Sender > - [Producer clientId=javaproducer11675630941604] Aborting producer > batches due to fatal error > org.apache.kafka.common.KafkaException: Unexpected error in > InitProducerIdResponse; The server experienced an unexpected error > when processing the request. > at > org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:1418) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1322) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575) > ~[kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:418) > [kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316) > [kafka-clients-3.2.3.jar:?] > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) >
Kafka issues related 3.2.3 version zookeeper less cluster
Hello All, We are facing some issues in the 3.2.3 kafka version cluster. We have implemented 3 brokers cluster on a single node server in the kubernetes environment. After every week are facing issues which are listed below. So, please help us to resolve these issues. Thanks to all in advance. *Issues we faced on brokers :* [2023-01-12 04:43:52,101] ERROR [KafkaApi-0] Unexpected error handling request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, correlationId=2) -- InitProducerIdRequestData(transactionalId=null, transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, clientId=dev-prp-clz-com-v2.1.105-MHM_CLZ_COM-DEV_123-3e0cf839-9c81-400c-94d2-975d330be75c-StreamThread-1-producer, correlationId=2), connectionId='10.233.80.1:9092-10.233.104.6:35952-208286', clientAddress=/10.233.104.6, principal=User:ANONYMOUS, listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, clientInformation=ClientInformation(softwareName=apache-kafka-java, softwareVersion=3.2.3), fromPrivilegedListener=true, principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3a6b516c]) (kafka.server.KafkaApis) org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request. [2023-01-12 04:43:52,159] WARN [RPC ProducerId Manager 0]: Had an unknown error from the controller, giving up. (kafka.coordinator.transaction.RPCProducerIdManager) [2023-01-12 04:43:52,159] ERROR [KafkaApi-0] Unexpected error handling request RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, correlationId=2) -- InitProducerIdRequestData(transactionalId=null, transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) with context RequestContext(header=RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, clientId=dev-prp-clz-com-v2.1.105-SCN_CLZ_COM_POST_PROCESS-DEV_123-4a353282-d8cf-4ac9-b2ea-a9c0e1163587-StreamThread-1-producer, correlationId=2), connectionId='10.233.80.1:9092-10.233.104.6:35946-208286', clientAddress=/10.233.104.6, principal=User:ANONYMOUS, listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, clientInformation=ClientInformation(softwareName=apache-kafka-java, softwareVersion=3.2.3), fromPrivilegedListener=true, principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@3042f6ba]) (kafka.server.KafkaApis) org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request. [2023-01-12 04:43:52,194] WARN [RPC ProducerId Manager 0]: Had an unknown error from the controller, giving up. (kafka.coordinator.transaction.RPCProducerIdManager) *Issues we faced on client :* 06:09:22.752 ERROR org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=javaproducer11675630941604] Aborting producer batches due to fatal error org.apache.kafka.common.KafkaException: Unexpected error in InitProducerIdResponse; The server experienced an unexpected error when processing the request. at org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:1418) ~[kafka-clients-3.2.3.jar:?] at org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1322) ~[kafka-clients-3.2.3.jar:?] at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) ~[kafka-clients-3.2.3.jar:?] at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583) ~[kafka-clients-3.2.3.jar:?] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575) ~[kafka-clients-3.2.3.jar:?] at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:418) [kafka-clients-3.2.3.jar:?] at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316) [kafka-clients-3.2.3.jar:?] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [kafka-clients-3.2.3.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] --- Thanks & Regards, Kunal Jadhav
