Hi Sönke, Thank you for the clarification – much appreciated.
Kind regards Pirow Engelbrecht | Senior Design Engineer Tel +27 12 678 9740 (ext. 9879) | Cell +27 63 148 3376 76 Regency Drive | Irene | Centurion | 0157<https://goo.gl/maps/v9ZbwjqpPyL2> [cid:image001.jpg@01D5FD28.3140B0E0]<https://etion.co.za> Facebook<https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr> | YouTube<https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A> | LinkedIn<https://www.linkedin.com/company/etionltd> | Twitter<https://twitter.com/Etionlimited> | Instagram<https://www.instagram.com/Etionlimited/> From: Sönke Liebau <soenke.lie...@opencore.com.INVALID> Sent: Wednesday, 18 March 2020 1:12 PM To: users@kafka.apache.org Subject: Re: Mirrormaker 2.0 and compacted topics Hi Pirow, records at the same offset as in the original topic is not possible for non compacted topics either, that is one of the major constraints of MirrorMaker. Compaction doesn't change offsets of messages, so in theory should behave the same as a non compacted topic, unless of course you miss messages as in your scenario. Additionally, as compaction is triggered individually by each cluster, the topics might contain different numbers of messages per key at any given time. This should not be relevant for a compaction use case, but can be good to keep in mind. Best regards, Sönke Pirow Engelbrecht <pirow.engelbre...@etion.co.za<mailto:pirow.engelbre...@etion.co.za>> schrieb am Mi., 18. März 2020, 12:01: > Hi Sönke, > > > > OK, thanks, so compacted topics is supported, but an exact replica (i.e. > Kafka records at the same offsets) of the original topic is not possible as > there is nothing to replicate. Is my understanding correct? > > > > Thanks > > > > *Pirow Engelbrecht *| Senior Design Engineer > *Tel* +27 12 678 9740 (ext. 9879) | *Cell* +27 63 148 3376 > > 76 Regency Drive | Irene | Centurion | 0157 > <https://goo.gl/maps/v9ZbwjqpPyL2<https://goo.gl/maps/v9ZbwjqpPyL2>> > > <https://etion.co.za<https://etion.co.za>> > > Facebook > <https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr<https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr>> > | > YouTube > <https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A<https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A>> > | > LinkedIn > <https://www.linkedin.com/company/etionltd<https://www.linkedin.com/company/etionltd>> > | Twitter > <https://twitter.com/Etionlimited<https://twitter.com/Etionlimited>> | > Instagram > <https://www.instagram.com/Etionlimited/<https://www.instagram.com/Etionlimited>> > > > > *From:* Sönke Liebau > <soenke.lie...@opencore.com.INVALID<mailto:soenke.lie...@opencore.com.INVALID>> > *Sent:* Wednesday, 18 March 2020 12:14 PM > *To:* users@kafka.apache.org<mailto:users@kafka.apache.org> > *Subject:* Re: Mirrormaker 2.0 and compacted topics > > > > Hi Pirow, > > as far as I understand MirrorMaker 2.0 will not treat compacted topics any > different from uncompacted topics. > > What that means for your scenario is that your replication may miss some > messages in the case of a long unavailability, if those messages were > compacted in the meantime. However, your replicated topic will always > receive the latest message per key so be "up-to-date" once MM is running > again. > > Not sure if that answers your question? > > Best regards, > Sönke > > > On Wed, 18 Mar 2020 at 09:18, Pirow Engelbrecht < > pirow.engelbre...@etion.co.za<mailto:pirow.engelbre...@etion.co.za>> wrote: > > > Hello, > > > > > > > > We’re currently trying to evaluate Mirrormaker 2.0 for future > > inter-cluster replication, replacing our bespoke replicator. I understand > > that Mirrormaker 2.0 documentation is only slated to be released in Kafka > > 2.5.0, but I was hoping that someone will know whether Mirrormaker 2.0 > can > > be used to replicate compacted topics? I am especially interested in the > > case where a cluster has been off-line for a few hours and have to catch > up > > via replication and the topic has been compacted since it was last seen > by > > the off-line cluster. > > > > > > > > Thanks > > > > > > > > *Pirow Engelbrecht *| Senior Design Engineer > > *Tel* +27 12 678 9740 (ext. 9879) | *Cell* +27 63 148 3376 > > > > 76 Regency Drive | Irene | Centurion | 0157 > > <https://goo.gl/maps/v9ZbwjqpPyL2<https://goo.gl/maps/v9ZbwjqpPyL2>> > > > > <https://etion.co.za<https://etion.co.za>> > > > > Facebook > > <https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr<https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr>> > > | > > YouTube > > <https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A<https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A>> > > | > > LinkedIn > > <https://www.linkedin.com/company/etionltd<https://www.linkedin.com/company/etionltd>> > > | Twitter > > <https://twitter.com/Etionlimited<https://twitter.com/Etionlimited>> | > > Instagram > > <https://www.instagram.com/Etionlimited/<https://www.instagram.com/Etionlimited/>> > > > > > > > > > -- > Sönke Liebau > Partner > Tel. +49 179 7940878 > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany >
