MirrorMaker 2.0 - Translating offsets for remote topics and consumer groups
Hi there, I'm currently exploring MM2 and having some trouble with the RemoteClusterUtils.translateOffsets() method. I have been successful in migrating a consumer group from the source cluster to the target cluster, but was wondering how I could migrate this consumer group back to the original source topic? It is my understanding that there isn't any checkpoint data being emitted for this consumer group since it is consuming from a mirrored topic in the target cluster. I'm currently getting an empty map since there isn't any checkpoint data for 'target.checkpoints.internal' in the source cluster. So, I was wondering how would I get these new translated offsets to migrate the consumer group back to the source cluster? Please let me know if my question was unclear or if you require further clarification! Appreciate the help. Thanks, Josh
Re: Steps & best-practices to upgrade Confluent Kafka 4.1x to 5.3x
You can follow below steps 1. set inter.broker.protocol.version=2.1.x and rolling restart kafka 2. Rolling upgrade the Kafka cluster to 2.5 - 3. rolling upgrade ZK cluster Validate the kafka . 4. set inter.broker.protocol.version= new version and rolling restart the Kafka On 8/18/20, 12:54 PM, "Rijo Roy" wrote: [External] Hi, I am a newbie in Kafka and would greatly appreciate if someone could help with best-practices and steps to upgrade to v5.3x. Below is my existing set-up: OS version: Ubuntu 16.04.6 LTS ZooKeeper version : 3.4.10 Kafka version : confluent-kafka-2.11 / 1.1.1-cp2 / v4.1.1 We need to upgrade our OS version to Ubuntu 18.04 LTS whose minimum requirement is to upgrade Kafka to v5.3x. Could someone please help me with the best-practices & steps for the upgrade.. Please let me know if you need any more information so that you could help me. Appreciate your help! Thanks & Regards, Rijo Roy This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo
Adding the user mailing list -- I think we should accepts votes on both lists for this special case, as it's not a technical decision. @Boyang: as mentioned by Bruno, can we maybe add black/white options for both proposals, too? I also agree that Design B is not ideal with regard to the Kafka logo. Would it be possible to change Design B accordingly? I am not a font expert, but the fonts in both design are different and I am wondering if there is an official Apache Kafka font that we should reuse to make sure that the logos align -- I would expect that both logos (including "Apache Kafka" and "Kafka Streams" names) will be used next to each other and it would look awkward if the font differs. -Matthias On 8/18/20 11:28 AM, Navinder Brar wrote: > Hi, > Thanks for the KIP, really like the idea. I am +1(non-binding) on A mainly > because I felt like you have to tilt your head to realize the otter's head in > B. > Regards,Navinder > > On Tuesday, 18 August, 2020, 11:44:20 pm IST, Guozhang Wang > wrote: > > I'm leaning towards design B primarily because it reminds me of the Firefox > logo which I like a lot. But I also share Adam's concern that it should > better not obscure the Kafka logo --- so if we can tweak a bit to fix it my > vote goes to B, otherwise A :) > > > Guozhang > > On Tue, Aug 18, 2020 at 9:48 AM Bruno Cadonna wrote: > >> Thanks for the KIP! >> >> I am +1 (non-binding) for A. >> >> I would also like to hear opinions whether the logo should be colorized >> or just black and white. >> >> Best, >> Bruno >> >> >> On 15.08.20 16:05, Adam Bellemare wrote: >>> I prefer Design B, but given that I missed the discussion thread, I think >>> it would be better without the Otter obscuring any part of the Kafka >> logo. >>> >>> On Thu, Aug 13, 2020 at 6:31 PM Boyang Chen >>> wrote: >>> Hello everyone, I would like to start a vote thread for KIP-657: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-657%3A+Add+Customized+Kafka+Streams+Logo This KIP is aiming to add a new logo for the Kafka Streams library. And >> we prepared two candidates with a cute otter. You could look up the KIP to find those logos. Please post your vote against these two customized logos. For example, I would vote for *design-A (binding)*. This vote thread shall be open for one week to collect enough votes to >> call for a winner. Still, feel free to post any question you may have >> regarding this KIP, thanks! >>> >> > > signature.asc Description: OpenPGP digital signature
Steps & best-practices to upgrade Confluent Kafka 4.1x to 5.3x
Hi, I am a newbie in Kafka and would greatly appreciate if someone could help with best-practices and steps to upgrade to v5.3x. Below is my existing set-up: OS version: Ubuntu 16.04.6 LTS ZooKeeper version : 3.4.10 Kafka version : confluent-kafka-2.11 / 1.1.1-cp2 / v4.1.1 We need to upgrade our OS version to Ubuntu 18.04 LTS whose minimum requirement is to upgrade Kafka to v5.3x. Could someone please help me with the best-practices & steps for the upgrade.. Please let me know if you need any more information so that you could help me. Appreciate your help! Thanks & Regards, Rijo Roy
CPU usage spikes and replication failure
Hi, Running Kafka 2.5.0_2.12. We run 3 identically configured kafka clusters (one in each region) and on one of them we've recently started having regular issues with spikes in CPU utilization lasting a few hours. In some cases we see large numbers of what look like replication failures, with the logs filled with: [2020-08-06 04:05:53,766] ERROR [ReplicaManager broker=0] Error processing append operation on partition __consumer_offsets-20 (kafka.server.ReplicaManager)org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(0) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-20 Once in this state the other brokers in the cluster also begin to fail with similar messages and our producers being to error. We've found that if we turn off all our producers temporarily the cluster quickly returns to normal, which indicates some additional load is being placed on the system, however all of our metrics show no increased message rates. Each cluster has 3 brokers, each with 4 CPU and 4 GiB memory, under normal load we see 1000-5000 message/s being produced, and we see maybe 10% utilization (the intention is to load this cluster more in future). During these periods of high utilization though we see CPU maxing out at 100% utilization for long periods of time. Searching around previous issues, I noticed https://issues.apache.org/jira/browse/KAFKA-4477 and https://issues.apache.org/jira/browse/KAFKA-6582 sounded similar, however both marked as fixed long before this release. I've attached a thread dump of one of the brokers during a high load period. It looks to my untrained eye like everything is just waiting on locks. Any assistance in further diagnosing this issue would be really appreciated. Our next step will probably be to completely rebuild this cluster, as we've only ever seen these issues on one of our instances. Regards, Toby 2020-08-18 17:12:33 Full thread dump OpenJDK 64-Bit Server VM (25.252-b09 mixed mode): "Attach Listener" #68 daemon prio=9 os_prio=0 tid=0x7fd8c0001000 nid=0x5a3 waiting on condition [0x] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Controller-0-to-broker-2-send-thread" #67 prio=5 os_prio=0 tid=0x7fd788079000 nid=0x58e waiting on condition [0x7fd75336c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00074d15f9b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:232) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) Locked ownable synchronizers: - None "executor-Heartbeat" #66 prio=5 os_prio=0 tid=0x7fd780003000 nid=0x58d waiting on condition [0x7fd7538fd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0007095aaa88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None "executor-Rebalance" #65 prio=5 os_prio=0 tid=0x7fd774003000 nid=0x58c waiting on condition [0x7fd7a01e6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0007095a27f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
