Hi Team, Hope you are doing great!!
This is Rakesh and I am managing Apache Kafka administration in an organization. Currently I am using Kafka version 2.4. I have few queries related to Kafka data replication among multiple Kafka clusters. 1. The first query is that, how to achieve 0 RPO or 0 data loss while replicating same Kafka topics among different Kafka clusters? Is Mirror Maker is the only solution? 2. If It is Mirror maker version 2 helps with same, is it possible to replicate/update only new records which we have received in the source Kafka cluster? *Explanation:* The query 2 is the most important use case which needs to get resolved, I can explain this scenario in depth with an example: "I have three Kafka clusters: *1. Source cluster (DC)* *2. Data Recovery (DR) Cluster* *3. Destination cluster*. Here I am replicating data from the source cluster (DC) to the data recovery(DR) cluster and destination cluster at the same time using Mirror Maker 2. Now if at any point of time if the destination cluster gets down it will lose the new records/data till the downtime and until it gets up. When it gets up it should heal up lost data from data recovery cluster, but: *Instead of replicating all the records from the source topic it should only replicate the delta records *(only new records which is there in the data recovery cluster )* to the destination cluster which can help to achieve Zero (0) RPO among multiple Kafka clusters.*" I hope the team would understand this scenario and that would be a great help if I get the solution on this. Also, we could connect remotely or on phone call to understand the depth of this or if there is any further queries/concerns on this use case. contact: *+91-7065990605, +91-8178237439 * email*: kmrac...@gmail.com <kmrac...@gmail.com>* google meet:* we can also connect on google meet.* On Sat, Mar 13, 2021, 11:33 AM <users-h...@kafka.apache.org> wrote: > Hi! This is the ezmlm program. I'm managing the > users@kafka.apache.org mailing list. > > I'm working for my owner, who can be reached > at users-ow...@kafka.apache.org. > > Acknowledgment: I have added the address > > kmrac...@gmail.com > > to the users mailing list. > > Welcome to users@kafka.apache.org! > > Please save this message so that you know the address you are > subscribed under, in case you later want to unsubscribe or change your > subscription address. > > > --- Administrative commands for the users list --- > > I can handle administrative requests automatically. Please > do not send them to the list address! Instead, send > your message to the correct command address: > > To subscribe to the list, send a message to: > <users-subscr...@kafka.apache.org> > > To remove your address from the list, send a message to: > <users-unsubscr...@kafka.apache.org> > > Send mail to the following for info and FAQ for this list: > <users-i...@kafka.apache.org> > <users-...@kafka.apache.org> > > Similar addresses exist for the digest list: > <users-digest-subscr...@kafka.apache.org> > <users-digest-unsubscr...@kafka.apache.org> > > To get messages 123 through 145 (a maximum of 100 per request), mail: > <users-get.123_...@kafka.apache.org> > > To get an index with subject and author for messages 123-456 , mail: > <users-index.123_...@kafka.apache.org> > > They are always returned as sets of 100, max 2000 per request, > so you'll actually get 100-499. > > To receive all messages with the same subject as message 12345, > send a short message to: > <users-thread.12...@kafka.apache.org> > > The messages should contain one line or word of text to avoid being > treated as sp@m, but I will ignore their content. > Only the ADDRESS you send to is important. > > You can start a subscription for an alternate address, > for example "john@host.domain", just add a hyphen and your > address (with '=' instead of '@') after the command word: > <users-subscribe-john=host.dom...@kafka.apache.org> > > To stop subscription for this address, mail: > <users-unsubscribe-john=host.dom...@kafka.apache.org> > > In both cases, I'll send a confirmation message to that address. When > you receive it, simply reply to it to complete your subscription. > > If despite following these instructions, you do not get the > desired results, please contact my owner at > users-ow...@kafka.apache.org. Please be patient, my owner is a > lot slower than I am ;-) > > --- Enclosed is a copy of the request I received. > > Return-Path: <kmrac...@gmail.com> > Received: (qmail 51351 invoked by uid 99); 13 Mar 2021 06:03:08 -0000 > Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) > (95.217.134.168) > by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2021 06:03:08 > +0000 > Received: from localhost (localhost [127.0.0.1]) > by spamproc1-he-fi.apache.org (ASF Mail Server at > spamproc1-he-fi.apache.org) with ESMTP id 5E07AC0340 > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > gmail....@kafka.apache.org>; Sat, 13 Mar 2021 06:03:07 +0000 (UTC) > X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org > X-Spam-Flag: NO > X-Spam-Score: 0.249 > X-Spam-Level: > X-Spam-Status: No, score=0.249 tagged_above=-999 required=6.31 > tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, > DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, > HTML_MESSAGE=0.2, > SPF_PASS=-0.001] autolearn=disabled > Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); > dkim=pass (2048-bit key) header.d=gmail.com > Received: from mx1-he-de.apache.org ([116.203.227.195]) > by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) > (amavisd-new, port 10024) > with ESMTP id UiffZgN6nw7z > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > gmail....@kafka.apache.org>; > Sat, 13 Mar 2021 06:03:07 +0000 (UTC) > Received-SPF: Pass (mailfrom) identity=mailfrom; > client-ip=2607:f8b0:4864:20::72c; helo=mail-qk1-x72c.google.com; > envelope-from=kmrac...@gmail.com; receiver=<UNKNOWN> > Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com > [IPv6:2607:f8b0:4864:20::72c]) > by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) > with ESMTPS id A558D7FB33 > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > gmail....@kafka.apache.org>; Sat, 13 Mar 2021 06:03:06 +0000 (UTC) > Received: by mail-qk1-x72c.google.com with SMTP id g185so26699302qkf.6 > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > gmail....@kafka.apache.org>; Fri, 12 Mar 2021 22:03:06 -0800 (PST) > DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; > d=gmail.com; s=20161025; > h=mime-version:from:date:message-id:subject:to; > bh=L3o3bidmj96UhVQkiPDbXpIWg+9CVEzOhM5hqoCVvcw=; > > b=MppgVNIsBEOBt4FpFiW7CO1wM/c3qUbtCJSOQ70ZoA7JZhCovZdrHT7UJzdijXNzUB > > xDGGHmoe6XJ+gCfjoiZM0f54qO/uWJlBFCDH92KwMTEBZ+J/gnF3OgdweWLwTQeco7Gt > > ZT/AVWE+60cGA+NlqyX6DH6ae775wXP6gJm0YFNad9DUYx0o3KqOd50eDOkpKuJyrLQ+ > > okVZsmsCVD5xNzSdwFNRA/jwrMcPiVU+P0SVbdUAsBion99qW0fg7RCrc1bPKdQTJ+VJ > > DF3aqTBc1n+lN9yhwRTgG/Txvt9miJ4LdtG+TzxK9a3EzA9f50L+yFUv5LVp/WTLFsUL > nPqw== > X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; > d=1e100.net; s=20161025; > h=x-gm-message-state:mime-version:from:date:message-id:subject:to; > bh=L3o3bidmj96UhVQkiPDbXpIWg+9CVEzOhM5hqoCVvcw=; > > b=fcqKFcdqZxgYHdLr0fIACEPc6YGKzWqFHhJlEEs+AtNdmD0I7lwibRQBl65kPrONC6 > > HyKD+9zg6/IYgdYiukhafCJAAdru2PmEYyRrNleick1UYy79+5GULJB2tDTH/8m6zIZb > > 6FE4zNwtVQmfgrYrLIA5G+B5h1RoU8vhncFhGdwaylQJN+aCp+g9eRTD9uMk6/BbsTvx > > UpQqOjhd69MkTKjWMCzhINdRuymSslF3S8crbfSV20s3/PIkLK0sCIe4dmdlrj0npc7Y > > 7KzrhlVzy6ufnPk0hC+xYsObufim5Acmgw6US+Rh0I8PrRcCKdhOx7EL2cwleVtPl95/ > drkw== > X-Gm-Message-State: > AOAM532oN9Urd4duX1v65+MvD+Q4p8Izy9RFxLCqQsQzKyfuv5QIEDJf > wCjTW5kNbBBCApHgj7motlpnqiLggPRfwu5nz13Th32G6mg= > X-Google-Smtp-Source: > ABdhPJxfMU9u852ODfxu7GW2ltEyIX/DU7yuYzwANMv6MXQcE2OKnpQQjDkJ0a9Gw5+JafjRxxu7ZbzfBd6ybXtiwbQ= > X-Received: by 2002:ae9:e80b:: with SMTP id > a11mr15793961qkg.65.1615615378992; > Fri, 12 Mar 2021 22:02:58 -0800 (PST) > MIME-Version: 1.0 > From: sanjay kumar <kmrac...@gmail.com> > Date: Sat, 13 Mar 2021 11:32:46 +0530 > Message-ID: < > cad2vzqepwfu_1ut5787iaghyrmqja3tlsn9ox+jfhoe6zwx...@mail.gmail.com> > Subject: kmracc89 mail confirmation > To: users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > gmail....@kafka.apache.org > Content-Type: multipart/alternative; > boundary="000000000000c2319905bd64c444" > >
