Re: Network partition leaves topic-partition leader as sole ISR despite min.isr=2 and producer acks=all settings

2024-08-14 Thread Sabit Nepal
d%20they%20also%20get%20removed > . > > On Mon, Aug 12, 2024 at 2:03 AM Sabit Nepal wrote: > > > Hello, > > > > We experienced a network partition in our kafka cluster which left 1 > broker > > unable to be reached by other brokers, but it could still reach

Network partition leaves topic-partition leader as sole ISR despite min.isr=2 and producer acks=all settings

2024-08-11 Thread Sabit Nepal
Hello, We experienced a network partition in our kafka cluster which left 1 broker unable to be reached by other brokers, but it could still reach our zookeeper cluster. When this occurred, a number of topic-partitions shrunk their ISR to just the impaired broker itself, halting progress on those

Streams configurations to optimize keeping standby tasks within acceptable recovery lag

2023-12-04 Thread Sabit Nepal
Hello, We recently started observing our streams application is taking hours to days to rebalance following events where streams consumers experience churn, such as code deployments. We increased the acceptable.recovery.lag property from the default of 10,000 to 100,000, however, we are still expe

Kafka Streams consumer application observing hotspotting during deployments

2023-11-01 Thread Sabit Nepal
Hello, We have a Kafka Streams consumer application running Kafka Streams 3.4, with our Kafka brokers running 2.6. This consumer application consumes from two topics with 250 partitions each, and we co-partition them to ensure each task is consuming from the same partitions in each topic. Some oth