Thanks Ryanne and Mickael I already suspected that something like this (i.e. current behaivior being a deliberate design decision) was the case and I just wanted to confirm my suspicions.
I will relay this internally On Fri, 11 Jun 2021, 18:55 Mickael Maison, <mickael.mai...@gmail.com> wrote: > Hi Matthew, > > If an administrator want to control topic creation, they should use > ACLs to prevents users creating topic. Relying on all applications to > not create topics is unlikely to succeed. > > Each refresh.topics.interval.seconds, MM2 checks if it needs to create > topics/partitions by comparing both clusters and also their previous > states. If your automation is able to create new topics on both > clusters, MM2 should detect them correctly and not attempt creating > any topics. If a topic does not exist on the target cluster, MM2 will > try creating it. If it fails, it will retry again at the next > refresh.topics.interval.seconds until the topic gets created. However, > it will also trigger a task reconfiguration each time which may have > an impact on your mirroring throughput. > > On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > > Matthew, I wonder what the expected behavior would be when topic-creation > > is disabled and MM is asked to replicate a topic that doesn't exist on > the > > target cluster? Maybe the task should fail at that point, or maybe it > > should replicate whatever it can? > > > > I think the current behavior is reasonable, esp considering precedent > from > > Connect and Streams, both of which actively create topics as needed. > > > > But I understand the motivation. Have they considered revoking topic > > creation permission using ACLs? > > > > Ryanne > > > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > Hello everyone, > > > > > > We have an interesting feature request from a client regarding having > the > > > property of automatic topic creation to be reflected in a MM2. > Specifically > > > the current behaviour where if you have automatic topic creation set to > > > false for the original Kafla cluster, MM2 configuration ignores this > which > > > means that if Kafka clients send messages to the MM2 then topics will > still > > > be automatically created on target cluster. The core problem here for > the > > > client is that our client wants to have complete control over how > topics > > > get created (i.e. with terraform setup scripts) and with the current > > > behaviour this is not possible. > > > > > > Of course this poses other problems if we did want to change the > behaviour > > > as stated earlier, i.e. if you update the configuration for the > original > > > Kafka cluster then you get into open questions about how to reflect > this > > > configuration onto the mirror maket (this is why its called "mirror"). > Is > > > making MM2 reflect that flag a feature that makes general or > alternately is > > > there another variation that makes more sense (i.e. having a separate > > > specific property rather than reusing the current automatic topic > creation > > > one). > > > > > > There is a currently existing issue on this at > > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 what > are > > > your thoughts on this? > > > > > > > > > -- > > > > > > Matthew de Detrich > > > > > > *Aiven Deutschland GmbH* > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > *m:* +491603708037 > > > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > > > >
