Hello Ryanne, Thank you for these clarifications. I am glad that the use case of “prefixless” replication is in-line with your vision for further development of MirrorMaker2; I forgot indeed that already KIP-382 mentioned that LegacyReplicationPolicy.
I have not found however any JIRA issue addressing specifically LegacyReplicationPolicy, and the main ticket for KIP-382 (KAFKA-7500) is already resolved. So what would be the next step you recommend I take? Should I file my feature request (rephrased to adjust for your comments)? Or is the need for the legacy policy already tracked somehow and we don’t have to open any new tickets for it? Marcin On 2020/03/11 14:20:44, Ryanne Dolan <ryannedo...@gmail.com> wrote: > Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not yet > released) which is similar to your PrefixlessReplicationPolicy and the > MigrationReplicationPolicy mentioned. I don't think these are hacky, though > maybe the documentation could better address what to do when a method > doesn't necessarily apply cleanly. > > Frankly, I anticipated needing to make changes to MM2 itself to accommodate > these sort of use-cases -- the fact that you got this working as-is is > encouraging :) > > Re the proposal, I think we could ship one or more of these replication > policies with MM2 to support prefixless replication. There may be slightly > different implementations depending on the use-case, i.e. "migration" from > an old cluster vs active/passive w/o renaming topics vs mimicking MM1 with > LegacyReplicationPolicy. Maybe they are all identical and we just need one? > > I like that use-cases are driving these requests rather than us try to > enumerate all possible ReplicationPolicy implementations. Given a > particular use-case and a successful application of a custom > ReplicationPolicy, I for one would support including it in Apache Kafka. > > Ryanne > > On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch <marcin.wiel...@gmail.com> > wrote: > > > Dear All, > > > > I would like to raise a feature request for ‘prefixless’ topic > > mirroring with MirrorMaker2 (see detailed description below). > > > > As after quite a few tries I am not 100% sure, could you please > > confirm that by now > > the ‘prefixless’ mirroring is not supported and that my feature > > request is valid? I would then file it in JIRA. > > > > Best regards, > > > > Marcin > > > > > > Feature Request > > > > We would like to use MirrorMaker2 as a DRP solution for a simple setup > > with two sites working in active-passive configuration > > where all topics are always mirrored in one direction only and the > > downstream topics keep their original names, i.e., without being > > prefixed by MirrorMaker2. > > > > After a few rounds of 'trial and error' we came up with a customized > > replication policy which, although hacky, seems to fulfill our need; > > it is presented in stackoverflow here: > > https://stackoverflow.com/a/60619233 > > and source code is available here: > > > > https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java > > > > This solution is 'hacky' in that it violates specification for the > > topicSource and upstreamTopic methods as expressed in javadoc of the > > ReplicationPolicy interface. > > > > That makes us suspect that MirrorMaker2 has been never meant to work > > in such a configuration and that our solution may stop > > working with future releases of Kafka. > > > > We would like then to ask for making the possibility of 'prefixless' > > topic mirroring in a simplified two-cluster setup a standard > > feature of Kafka Mirror Maker 2. > > >
