Re: MirrorMaker2 Question

Hi Harry, Thanks for your question! There were negative lag/data loss bugs fixed some time back [1, 2] so check what version of the MirrorMaker connectors you're using. I would recommend using the latest version of MM2 (3.8.0) if possible. After upgrading you will need to clear the checkpoints t

MirrorMaker2 Question

Hi everyone, Does anyone know if it is possible for the Checkpoint connector to be ahead of the Source connector? For context, I have a target Kafka cluster that is reporting negative lag because the log end offset is 0 but the consumer group offset is 100. I have a tenuous theory that this co

producer Java security.protocol

In CLI, in my krb-client.properties files I can do security.protocol=SASL_SSL sasl.mechanism=GSSAPI And I invoke echo "bar" | kafka-console-producer.sh --producer.config krb-client.properties it works great. I have a custom Java client but I am not sure how I can incorporate these settings. How

Re: Facing large election time in Kafka

Hey Atul Leadership information is propagated to the rest of the brokers by the controller. In Zk mode, the controller may take a long time to start up because it needs to fetch all state from Zookeeper. Rolling restart makes it worse because you may end up having controller re-election multiple t

Re: Facing issue while producing

Hi Vikram. Can you share you code snippet? Are you using compression on producer/broker side? On Friday, August 23, 2024, Vikram Singh wrote: > Hello, > > I am facing issues while producing messages on kafka topics. I am facing > this issue randomly. Please help me by referring to the logs below

Re: Facing large election time in Kafka

Hi Akash, thanks for the quick reply. By the term leader i mean partition leader. I agree with you as upgrading to Kraft based cluster will help in reducing the partitions leader election time, but as we already have Zookeeper based Kafka cluster upgrading to Kraft based cluster will be bit chal

Re: Facing large election time in Kafka

For Kafka with Zookeeper the recovery time is proportional to the number of partitions in the cluster. So theoretically speaking the behaviour is consistent - it will take time. Kraft based Kafka clusters (since Kafka v3.3) are much much better with clusters with a large number of partitions such a

Facing issue while producing

Hello, I am facing issues while producing messages on kafka topics. I am facing this issue randomly. Please help me by referring to the logs below. Logs :- 1). ERROR [ReplicaManager broker=0] Error processing append operation on partition MHM_CLZ_COM-AWS_123-7 (kafka.server.ReplicaManager) org.a

Re: Facing large election time in Kafka

HI Atul you use the word 'leader'. You mean the 'controller'? Or you referring to the leader for each of the partitions? On Fri, Aug 23, 2024 at 7:44 AM Atul Sharma wrote: > Hi, > We are currently facing a prolonged leader election time, approx 2 mins, in > a Kafka cluster (version 2.8.2) that i

Re: Handling load distribution evenly, with ordering

In your producer application, you could write a logic for key1 & key2 to be produced to separate topics with appropriate number of partitions for that key according to your throughput requirements. For the consumer application you can configure it to consumer from multiple topics or can have instan