Hi Team, Hope you are doing great!!
This is Rakesh and I am managing Apache Kafka administration in an organization. Currently I am using Kafka version 2.4. I have few queries related to Kafka data replication among multiple Kafka clusters. 1. The first query is that, how to achieve 0 RPO or 0 data loss while replicating same Kafka topics among different Kafka clusters? Is Mirror Maker is the only solution? 2. If It is Mirror maker version 2 helps with same, is it possible to replicate/update only new records which we have received in the source Kafka cluster? *Explanation:* The query 2 is the most important use case which needs to get resolved, I can explain this scenario in depth with an example: "I have three Kafka clusters: *1. Source cluster (DC)* *2. Data Recovery (DR) Cluster* *3. Destination cluster*. Here I am replicating data from the source cluster (DC) to the data recovery(DR) cluster and destination cluster at the same time using Mirror Maker 2. Now if at any point of time if the destination cluster gets down it will lose the new records/data till the downtime and until it gets up. When it gets up it should heal up lost data from data recovery cluster, but: *Instead of replicating all the records from the source topic it should only replicate the delta records *(only new records which is there in the data recovery cluster )* to the destination cluster which can help to achieve Zero (0) RPO among multiple Kafka clusters.*" I hope the team would understand this scenario and that would be a great help if I get the solution on this. Also, we could connect remotely or on phone call to understand the depth of this or if there is any further queries/concerns on this use case. contact: *+91-7065990605, +91-8178237439 * email*: [email protected] <[email protected]>* google meet:* we can also connect on google meet.* On Sat, Mar 13, 2021, 11:33 AM <[email protected]> wrote: > Hi! This is the ezmlm program. I'm managing the > [email protected] mailing list. > > I'm working for my owner, who can be reached > at [email protected]. > > Acknowledgment: I have added the address > > [email protected] > > to the users mailing list. > > Welcome to [email protected]! > > Please save this message so that you know the address you are > subscribed under, in case you later want to unsubscribe or change your > subscription address. > > > --- Administrative commands for the users list --- > > I can handle administrative requests automatically. Please > do not send them to the list address! Instead, send > your message to the correct command address: > > To subscribe to the list, send a message to: > <[email protected]> > > To remove your address from the list, send a message to: > <[email protected]> > > Send mail to the following for info and FAQ for this list: > <[email protected]> > <[email protected]> > > Similar addresses exist for the digest list: > <[email protected]> > <[email protected]> > > To get messages 123 through 145 (a maximum of 100 per request), mail: > <[email protected]> > > To get an index with subject and author for messages 123-456 , mail: > <[email protected]> > > They are always returned as sets of 100, max 2000 per request, > so you'll actually get 100-499. > > To receive all messages with the same subject as message 12345, > send a short message to: > <[email protected]> > > The messages should contain one line or word of text to avoid being > treated as sp@m, but I will ignore their content. > Only the ADDRESS you send to is important. > > You can start a subscription for an alternate address, > for example "[email protected]", just add a hyphen and your > address (with '=' instead of '@') after the command word: > <[email protected]> > > To stop subscription for this address, mail: > <[email protected]> > > In both cases, I'll send a confirmation message to that address. When > you receive it, simply reply to it to complete your subscription. > > If despite following these instructions, you do not get the > desired results, please contact my owner at > [email protected]. Please be patient, my owner is a > lot slower than I am ;-) > > --- Enclosed is a copy of the request I received. > > Return-Path: <[email protected]> > Received: (qmail 51351 invoked by uid 99); 13 Mar 2021 06:03:08 -0000 > Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) > (95.217.134.168) > by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2021 06:03:08 > +0000 > Received: from localhost (localhost [127.0.0.1]) > by spamproc1-he-fi.apache.org (ASF Mail Server at > spamproc1-he-fi.apache.org) with ESMTP id 5E07AC0340 > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > [email protected]>; Sat, 13 Mar 2021 06:03:07 +0000 (UTC) > X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org > X-Spam-Flag: NO > X-Spam-Score: 0.249 > X-Spam-Level: > X-Spam-Status: No, score=0.249 tagged_above=-999 required=6.31 > tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, > DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, > HTML_MESSAGE=0.2, > SPF_PASS=-0.001] autolearn=disabled > Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); > dkim=pass (2048-bit key) header.d=gmail.com > Received: from mx1-he-de.apache.org ([116.203.227.195]) > by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) > (amavisd-new, port 10024) > with ESMTP id UiffZgN6nw7z > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > [email protected]>; > Sat, 13 Mar 2021 06:03:07 +0000 (UTC) > Received-SPF: Pass (mailfrom) identity=mailfrom; > client-ip=2607:f8b0:4864:20::72c; helo=mail-qk1-x72c.google.com; > [email protected]; receiver=<UNKNOWN> > Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com > [IPv6:2607:f8b0:4864:20::72c]) > by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) > with ESMTPS id A558D7FB33 > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > [email protected]>; Sat, 13 Mar 2021 06:03:06 +0000 (UTC) > Received: by mail-qk1-x72c.google.com with SMTP id g185so26699302qkf.6 > for <users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > [email protected]>; Fri, 12 Mar 2021 22:03:06 -0800 (PST) > DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; > d=gmail.com; s=20161025; > h=mime-version:from:date:message-id:subject:to; > bh=L3o3bidmj96UhVQkiPDbXpIWg+9CVEzOhM5hqoCVvcw=; > > b=MppgVNIsBEOBt4FpFiW7CO1wM/c3qUbtCJSOQ70ZoA7JZhCovZdrHT7UJzdijXNzUB > > xDGGHmoe6XJ+gCfjoiZM0f54qO/uWJlBFCDH92KwMTEBZ+J/gnF3OgdweWLwTQeco7Gt > > ZT/AVWE+60cGA+NlqyX6DH6ae775wXP6gJm0YFNad9DUYx0o3KqOd50eDOkpKuJyrLQ+ > > okVZsmsCVD5xNzSdwFNRA/jwrMcPiVU+P0SVbdUAsBion99qW0fg7RCrc1bPKdQTJ+VJ > > DF3aqTBc1n+lN9yhwRTgG/Txvt9miJ4LdtG+TzxK9a3EzA9f50L+yFUv5LVp/WTLFsUL > nPqw== > X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; > d=1e100.net; s=20161025; > h=x-gm-message-state:mime-version:from:date:message-id:subject:to; > bh=L3o3bidmj96UhVQkiPDbXpIWg+9CVEzOhM5hqoCVvcw=; > > b=fcqKFcdqZxgYHdLr0fIACEPc6YGKzWqFHhJlEEs+AtNdmD0I7lwibRQBl65kPrONC6 > > HyKD+9zg6/IYgdYiukhafCJAAdru2PmEYyRrNleick1UYy79+5GULJB2tDTH/8m6zIZb > > 6FE4zNwtVQmfgrYrLIA5G+B5h1RoU8vhncFhGdwaylQJN+aCp+g9eRTD9uMk6/BbsTvx > > UpQqOjhd69MkTKjWMCzhINdRuymSslF3S8crbfSV20s3/PIkLK0sCIe4dmdlrj0npc7Y > > 7KzrhlVzy6ufnPk0hC+xYsObufim5Acmgw6US+Rh0I8PrRcCKdhOx7EL2cwleVtPl95/ > drkw== > X-Gm-Message-State: > AOAM532oN9Urd4duX1v65+MvD+Q4p8Izy9RFxLCqQsQzKyfuv5QIEDJf > wCjTW5kNbBBCApHgj7motlpnqiLggPRfwu5nz13Th32G6mg= > X-Google-Smtp-Source: > ABdhPJxfMU9u852ODfxu7GW2ltEyIX/DU7yuYzwANMv6MXQcE2OKnpQQjDkJ0a9Gw5+JafjRxxu7ZbzfBd6ybXtiwbQ= > X-Received: by 2002:ae9:e80b:: with SMTP id > a11mr15793961qkg.65.1615615378992; > Fri, 12 Mar 2021 22:02:58 -0800 (PST) > MIME-Version: 1.0 > From: sanjay kumar <[email protected]> > Date: Sat, 13 Mar 2021 11:32:46 +0530 > Message-ID: < > cad2vzqepwfu_1ut5787iaghyrmqja3tlsn9ox+jfhoe6zwx...@mail.gmail.com> > Subject: kmracc89 mail confirmation > To: users-sc.1615565903.ghnjhkgahmehopopiofo-kmracc89= > [email protected] > Content-Type: multipart/alternative; > boundary="000000000000c2319905bd64c444" > >
