Maybe it would be nice if it was possible to hook into or extend Admin client interactions, to allow for custom logic supporting use cases such as this. Scenarios where topic/resource management is centralized are probably not uncommon.
-- Igor On Sat, Jun 12, 2021, at 9:42 AM, Matthew de Detrich wrote: > Thanks Ryanne and Mickael > > I already suspected that something like this (i.e. current behaivior being > a deliberate design decision) was the case and I just wanted to confirm my > suspicions. > > I will relay this internally > > On Fri, 11 Jun 2021, 18:55 Mickael Maison, <mickael.mai...@gmail.com> wrote: > > > Hi Matthew, > > > > If an administrator want to control topic creation, they should use > > ACLs to prevents users creating topic. Relying on all applications to > > not create topics is unlikely to succeed. > > > > Each refresh.topics.interval.seconds, MM2 checks if it needs to create > > topics/partitions by comparing both clusters and also their previous > > states. If your automation is able to create new topics on both > > clusters, MM2 should detect them correctly and not attempt creating > > any topics. If a topic does not exist on the target cluster, MM2 will > > try creating it. If it fails, it will retry again at the next > > refresh.topics.interval.seconds until the topic gets created. However, > > it will also trigger a task reconfiguration each time which may have > > an impact on your mirroring throughput. > > > > On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > > > > Matthew, I wonder what the expected behavior would be when topic-creation > > > is disabled and MM is asked to replicate a topic that doesn't exist on > > the > > > target cluster? Maybe the task should fail at that point, or maybe it > > > should replicate whatever it can? > > > > > > I think the current behavior is reasonable, esp considering precedent > > from > > > Connect and Streams, both of which actively create topics as needed. > > > > > > But I understand the motivation. Have they considered revoking topic > > > creation permission using ACLs? > > > > > > Ryanne > > > > > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > Hello everyone, > > > > > > > > We have an interesting feature request from a client regarding having > > the > > > > property of automatic topic creation to be reflected in a MM2. > > Specifically > > > > the current behaviour where if you have automatic topic creation set to > > > > false for the original Kafla cluster, MM2 configuration ignores this > > which > > > > means that if Kafka clients send messages to the MM2 then topics will > > still > > > > be automatically created on target cluster. The core problem here for > > the > > > > client is that our client wants to have complete control over how > > topics > > > > get created (i.e. with terraform setup scripts) and with the current > > > > behaviour this is not possible. > > > > > > > > Of course this poses other problems if we did want to change the > > behaviour > > > > as stated earlier, i.e. if you update the configuration for the > > original > > > > Kafka cluster then you get into open questions about how to reflect > > this > > > > configuration onto the mirror maket (this is why its called "mirror"). > > Is > > > > making MM2 reflect that flag a feature that makes general or > > alternately is > > > > there another variation that makes more sense (i.e. having a separate > > > > specific property rather than reusing the current automatic topic > > creation > > > > one). > > > > > > > > There is a currently existing issue on this at > > > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 what > > are > > > > your thoughts on this? > > > > > > > > > > > > -- > > > > > > > > Matthew de Detrich > > > > > > > > *Aiven Deutschland GmbH* > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > *m:* +491603708037 > > > > > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > > > > > > >
