How do we usually handle Node disconnected issue for kafka producer
Hi, I am using a Kafka producer java client by vert.x framework. https://vertx.io/docs/apidocs/io/vertx/kafka/client/producer/KafkaProducer.html There is a producer setting in kafka: connections.max.idle.ms = 54 So if there are no records to produce then after 9 minutes I get this in my logs: [kafka-producer-network-thread | RecordProducer] [NetworkClient.java:977] - [Producer clientId=RecordProducer] Node -1 disconnected. What it looks like is the Kafka producer object I had created has lost its connection due to this setting. What are my options to ensure that Kafka producer client does not close idle connections or reconnects or keeps alive even when no records to produce arrive for a long time? Thanks Sachin
Re: How do we usually handle Node disconnected issue for kafka producer
Hi Sachin, Why do you want to change the default settings? If the connection is open and unused, then it is fair to close the connection after the timeout and reopen it when required. On Fri, May 3, 2024 at 1:06 PM Sachin Mittal wrote: > Hi, > I am using a Kafka producer java client by vert.x framework. > > https://vertx.io/docs/apidocs/io/vertx/kafka/client/producer/KafkaProducer.html > > There is a producer setting in kafka: > connections.max.idle.ms = 54 > > So if there are no records to produce then after 9 minutes I get this in my > logs: > [kafka-producer-network-thread | RecordProducer] [NetworkClient.java:977] - > [Producer clientId=RecordProducer] Node -1 disconnected. > > What it looks like is the Kafka producer object I had created has lost its > connection due to this setting. > > What are my options to ensure that Kafka producer client does not close > idle connections or reconnects or keeps alive even when no records to > produce arrive for a long time? > > Thanks > Sachin >
Re: How do we usually handle Node disconnected issue for kafka producer
Hi Kamal, I understand this, however the connections are maintained by a vertx Kafka client and I am not able to find a way to catch the closed connection and reopen it. Would setting the connections.max.idle.ms = -1 or max int/long help here. Thanks Sachin On Sat, 4 May 2024 at 11:06 PM, Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Sachin, > > Why do you want to change the default settings? If the connection is open > and unused, > then it is fair to close the connection after the timeout and reopen it > when required. > > On Fri, May 3, 2024 at 1:06 PM Sachin Mittal wrote: > > > Hi, > > I am using a Kafka producer java client by vert.x framework. > > > > > https://vertx.io/docs/apidocs/io/vertx/kafka/client/producer/KafkaProducer.html > > > > There is a producer setting in kafka: > > connections.max.idle.ms = 54 > > > > So if there are no records to produce then after 9 minutes I get this in > my > > logs: > > [kafka-producer-network-thread | RecordProducer] > [NetworkClient.java:977] - > > [Producer clientId=RecordProducer] Node -1 disconnected. > > > > What it looks like is the Kafka producer object I had created has lost > its > > connection due to this setting. > > > > What are my options to ensure that Kafka producer client does not close > > idle connections or reconnects or keeps alive even when no records to > > produce arrive for a long time? > > > > Thanks > > Sachin > > >
