Re: Apache zookeeper going down every 168 hours

2024-05-03 Thread Haruki Okada
Thanks for sharing logs. Kafka has a mechanism to mark log dir as "failed" when IOException happens on I/O operations, and it will shut down when all log dirs have marked as failed. (Kafka allows to set multiple log dirs for JBOD) >From server.log, we can see that the Kafka broker shut down

Re: Apache zookeeper going down every 168 hours

2024-05-03 Thread Yogeshkumar Annadurai
Hello, We see timeout error in server.log log files and properties files are attached for your reference regards Yogeshkumar A On Sat, May 4, 2024 at 5:27 AM Haruki Okada wrote: > Hi. > > log.retention shouldn't be related to the phenomenon. > Sounds like we should understand the situation

Re: Apache zookeeper going down every 168 hours

2024-05-03 Thread Haruki Okada
Hi. log.retention shouldn't be related to the phenomenon. Sounds like we should understand the situation more precisely to answer. > apache zookeeper connection is going down automatically How did you confirm this? On ZooKeeper log? Also, did you see any logs on Kafka side? (on stdout or

Apache zookeeper going down every 168 hours

2024-05-03 Thread Yogeshkumar Annadurai
Hello, We are using Apache kakfa in a development environment, where apache zookeeper connection is going down automatically every 168 hours. we observed that, log.retention.hours is set as 168 hours (7 days). I would like to understand the configuration for this kind of scenario (automatic

Re: Failed to initialize processor KSTREAM-AGGREGATE-0000000001

2024-05-03 Thread Matthias J. Sax
Can you file a ticket for it: https://issues.apache.org/jira/browse/KAFKA On 5/3/24 3:34 AM, Penumarthi Durga Prasad Chowdary wrote: Kafka versions 3.5.1 and 3.7.0, we're still encountering persistent issues. The Kafka Streams library is aligned with these Kafka versions. Upon analysis of the

Re: Failed to initialize processor KSTREAM-AGGREGATE-0000000001

2024-05-03 Thread Penumarthi Durga Prasad Chowdary
Kafka versions 3.5.1 and 3.7.0, we're still encountering persistent issues. The Kafka Streams library is aligned with these Kafka versions. Upon analysis of the logs, it seems that the problem may occur when a Kafka node disconnects from Kafka Streams processes. This suspicion is supported by the

How do we usually handle Node disconnected issue for kafka producer

2024-05-03 Thread Sachin Mittal
Hi, I am using a Kafka producer java client by vert.x framework. https://vertx.io/docs/apidocs/io/vertx/kafka/client/producer/KafkaProducer.html There is a producer setting in kafka: connections.max.idle.ms = 54 So if there are no records to produce then after 9 minutes I get this in my