Hi Matthew, If an administrator want to control topic creation, they should use ACLs to prevents users creating topic. Relying on all applications to not create topics is unlikely to succeed.
Each refresh.topics.interval.seconds, MM2 checks if it needs to create topics/partitions by comparing both clusters and also their previous states. If your automation is able to create new topics on both clusters, MM2 should detect them correctly and not attempt creating any topics. If a topic does not exist on the target cluster, MM2 will try creating it. If it fails, it will retry again at the next refresh.topics.interval.seconds until the topic gets created. However, it will also trigger a task reconfiguration each time which may have an impact on your mirroring throughput. On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > > Matthew, I wonder what the expected behavior would be when topic-creation > is disabled and MM is asked to replicate a topic that doesn't exist on the > target cluster? Maybe the task should fail at that point, or maybe it > should replicate whatever it can? > > I think the current behavior is reasonable, esp considering precedent from > Connect and Streams, both of which actively create topics as needed. > > But I understand the motivation. Have they considered revoking topic > creation permission using ACLs? > > Ryanne > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > <matthew.dedetr...@aiven.io.invalid> wrote: > > > Hello everyone, > > > > We have an interesting feature request from a client regarding having the > > property of automatic topic creation to be reflected in a MM2. Specifically > > the current behaviour where if you have automatic topic creation set to > > false for the original Kafla cluster, MM2 configuration ignores this which > > means that if Kafka clients send messages to the MM2 then topics will still > > be automatically created on target cluster. The core problem here for the > > client is that our client wants to have complete control over how topics > > get created (i.e. with terraform setup scripts) and with the current > > behaviour this is not possible. > > > > Of course this poses other problems if we did want to change the behaviour > > as stated earlier, i.e. if you update the configuration for the original > > Kafka cluster then you get into open questions about how to reflect this > > configuration onto the mirror maket (this is why its called "mirror"). Is > > making MM2 reflect that flag a feature that makes general or alternately is > > there another variation that makes more sense (i.e. having a separate > > specific property rather than reusing the current automatic topic creation > > one). > > > > There is a currently existing issue on this at > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 what are > > your thoughts on this? > > > > > > -- > > > > Matthew de Detrich > > > > *Aiven Deutschland GmbH* > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > *m:* +491603708037 > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > >
