Re: MM2 taking into consideration automatic topic creation property from original cluster
Hi Omnia Maybe - I'll look forward to reading the KIP when it's submitted. When I researched the feature request from our client I found KIP-158 which added a bunch of topic auto-creation controls to the Kafka Connect framework in version 2.6. It seemed to me like a convenient solution would be to extend MM2 to use this rather than going directly to AdminClient. To Ryanne & Mickael's point about controlling this using ACLs, I don't like this so much because if MM2 will constantly try and create topics every so often (period defined by refresh.topics.interval.seconds) then this will add unnecessary overhead to the service and create a lot of noise in error and audit logs. I would prefer to be able to turn off topic auto-creation completely. For Kafka Streams internal topics, changelogs etc, it makes sense that these are always auto-created. These are implementation details and the user may not even care or need to know that they exist (but if they aren't there then strange issues will surely manifest) Topics created by MM2 are "owned" by the users, not the MM2 service. So I think there's precedent for users to be able to control auto-creation, just like they can with Connect and with Kafka in general. Dave On Mon, Jun 14, 2021 at 2:32 PM Omnia Ibrahim wrote: > Hi, > I am in the middle of writing a new KIP to introduced a new interface for > topic updates/creation/description that will allow MM2 to either uses the > default behaviour which uses Kafka AdminClient to create/update topics or > to use a customised one that integrates with the user's ecosystem where > they can call their Kafka resource management service to create a topic or > update the configuration of the topic. Would this be a possible solution > for your use case where you can provide a custom implementation that works > with your ecosystem to create the topic? > > -- > Omnia > > On Mon, Jun 14, 2021 at 2:25 PM Igor Soarez wrote: > > > Maybe it would be nice if it was possible to hook into or extend Admin > > client interactions, to allow for custom logic supporting use cases such > as > > this. > > Scenarios where topic/resource management is centralized are probably not > > uncommon. > > > > -- > > Igor > > > > On Sat, Jun 12, 2021, at 9:42 AM, Matthew de Detrich wrote: > > > Thanks Ryanne and Mickael > > > > > > I already suspected that something like this (i.e. current behaivior > > being > > > a deliberate design decision) was the case and I just wanted to confirm > > my > > > suspicions. > > > > > > I will relay this internally > > > > > > On Fri, 11 Jun 2021, 18:55 Mickael Maison, > > wrote: > > > > > > > Hi Matthew, > > > > > > > > If an administrator want to control topic creation, they should use > > > > ACLs to prevents users creating topic. Relying on all applications to > > > > not create topics is unlikely to succeed. > > > > > > > > Each refresh.topics.interval.seconds, MM2 checks if it needs to > create > > > > topics/partitions by comparing both clusters and also their previous > > > > states. If your automation is able to create new topics on both > > > > clusters, MM2 should detect them correctly and not attempt creating > > > > any topics. If a topic does not exist on the target cluster, MM2 will > > > > try creating it. If it fails, it will retry again at the next > > > > refresh.topics.interval.seconds until the topic gets created. > However, > > > > it will also trigger a task reconfiguration each time which may have > > > > an impact on your mirroring throughput. > > > > > > > > On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan > > > > wrote: > > > > > > > > > > Matthew, I wonder what the expected behavior would be when > > topic-creation > > > > > is disabled and MM is asked to replicate a topic that doesn't exist > > on > > > > the > > > > > target cluster? Maybe the task should fail at that point, or maybe > it > > > > > should replicate whatever it can? > > > > > > > > > > I think the current behavior is reasonable, esp considering > precedent > > > > from > > > > > Connect and Streams, both of which actively create topics as > needed. > > > > > > > > > > But I understand the motivation. Have they considered revoking > topic > > > > > creation permission using ACLs? > > > > > > > > > > Ryanne > > > > > > > > > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > > > > > wrote: > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > We have an interesting feature request from a client regarding > > having > > > > the > > > > > > property of automatic topic creation to be reflected in a MM2. > > > > Specifically > > > > > > the current behaviour where if you have automatic topic creation > > set to > > > > > > false for the original Kafla cluster, MM2 configuration ignores > > this > > > > which > > > > > > means that if Kafka clients send messages to the MM2 then topics > > will > > > > still > > > > > > be automatically created on target cluster. The core problem here > > for > > > > the > > > > > >
Re: MM2 taking into consideration automatic topic creation property from original cluster
Hi, I am in the middle of writing a new KIP to introduced a new interface for topic updates/creation/description that will allow MM2 to either uses the default behaviour which uses Kafka AdminClient to create/update topics or to use a customised one that integrates with the user's ecosystem where they can call their Kafka resource management service to create a topic or update the configuration of the topic. Would this be a possible solution for your use case where you can provide a custom implementation that works with your ecosystem to create the topic? -- Omnia On Mon, Jun 14, 2021 at 2:25 PM Igor Soarez wrote: > Maybe it would be nice if it was possible to hook into or extend Admin > client interactions, to allow for custom logic supporting use cases such as > this. > Scenarios where topic/resource management is centralized are probably not > uncommon. > > -- > Igor > > On Sat, Jun 12, 2021, at 9:42 AM, Matthew de Detrich wrote: > > Thanks Ryanne and Mickael > > > > I already suspected that something like this (i.e. current behaivior > being > > a deliberate design decision) was the case and I just wanted to confirm > my > > suspicions. > > > > I will relay this internally > > > > On Fri, 11 Jun 2021, 18:55 Mickael Maison, > wrote: > > > > > Hi Matthew, > > > > > > If an administrator want to control topic creation, they should use > > > ACLs to prevents users creating topic. Relying on all applications to > > > not create topics is unlikely to succeed. > > > > > > Each refresh.topics.interval.seconds, MM2 checks if it needs to create > > > topics/partitions by comparing both clusters and also their previous > > > states. If your automation is able to create new topics on both > > > clusters, MM2 should detect them correctly and not attempt creating > > > any topics. If a topic does not exist on the target cluster, MM2 will > > > try creating it. If it fails, it will retry again at the next > > > refresh.topics.interval.seconds until the topic gets created. However, > > > it will also trigger a task reconfiguration each time which may have > > > an impact on your mirroring throughput. > > > > > > On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan > > > wrote: > > > > > > > > Matthew, I wonder what the expected behavior would be when > topic-creation > > > > is disabled and MM is asked to replicate a topic that doesn't exist > on > > > the > > > > target cluster? Maybe the task should fail at that point, or maybe it > > > > should replicate whatever it can? > > > > > > > > I think the current behavior is reasonable, esp considering precedent > > > from > > > > Connect and Streams, both of which actively create topics as needed. > > > > > > > > But I understand the motivation. Have they considered revoking topic > > > > creation permission using ACLs? > > > > > > > > Ryanne > > > > > > > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > > > > wrote: > > > > > > > > > Hello everyone, > > > > > > > > > > We have an interesting feature request from a client regarding > having > > > the > > > > > property of automatic topic creation to be reflected in a MM2. > > > Specifically > > > > > the current behaviour where if you have automatic topic creation > set to > > > > > false for the original Kafla cluster, MM2 configuration ignores > this > > > which > > > > > means that if Kafka clients send messages to the MM2 then topics > will > > > still > > > > > be automatically created on target cluster. The core problem here > for > > > the > > > > > client is that our client wants to have complete control over how > > > topics > > > > > get created (i.e. with terraform setup scripts) and with the > current > > > > > behaviour this is not possible. > > > > > > > > > > Of course this poses other problems if we did want to change the > > > behaviour > > > > > as stated earlier, i.e. if you update the configuration for the > > > original > > > > > Kafka cluster then you get into open questions about how to reflect > > > this > > > > > configuration onto the mirror maket (this is why its called > "mirror"). > > > Is > > > > > making MM2 reflect that flag a feature that makes general or > > > alternately is > > > > > there another variation that makes more sense (i.e. having a > separate > > > > > specific property rather than reusing the current automatic topic > > > creation > > > > > one). > > > > > > > > > > There is a currently existing issue on this at > > > > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > > > > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 > what > > > are > > > > > your thoughts on this? > > > > > > > > > > > > > > > -- > > > > > > > > > > Matthew de Detrich > > > > > > > > > > *Aiven Deutschland GmbH* > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > *m:* +491603708037 > > > > > > > > > > *w:*
Re: MM2 taking into consideration automatic topic creation property from original cluster
Maybe it would be nice if it was possible to hook into or extend Admin client interactions, to allow for custom logic supporting use cases such as this. Scenarios where topic/resource management is centralized are probably not uncommon. -- Igor On Sat, Jun 12, 2021, at 9:42 AM, Matthew de Detrich wrote: > Thanks Ryanne and Mickael > > I already suspected that something like this (i.e. current behaivior being > a deliberate design decision) was the case and I just wanted to confirm my > suspicions. > > I will relay this internally > > On Fri, 11 Jun 2021, 18:55 Mickael Maison, wrote: > > > Hi Matthew, > > > > If an administrator want to control topic creation, they should use > > ACLs to prevents users creating topic. Relying on all applications to > > not create topics is unlikely to succeed. > > > > Each refresh.topics.interval.seconds, MM2 checks if it needs to create > > topics/partitions by comparing both clusters and also their previous > > states. If your automation is able to create new topics on both > > clusters, MM2 should detect them correctly and not attempt creating > > any topics. If a topic does not exist on the target cluster, MM2 will > > try creating it. If it fails, it will retry again at the next > > refresh.topics.interval.seconds until the topic gets created. However, > > it will also trigger a task reconfiguration each time which may have > > an impact on your mirroring throughput. > > > > On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan > > wrote: > > > > > > Matthew, I wonder what the expected behavior would be when topic-creation > > > is disabled and MM is asked to replicate a topic that doesn't exist on > > the > > > target cluster? Maybe the task should fail at that point, or maybe it > > > should replicate whatever it can? > > > > > > I think the current behavior is reasonable, esp considering precedent > > from > > > Connect and Streams, both of which actively create topics as needed. > > > > > > But I understand the motivation. Have they considered revoking topic > > > creation permission using ACLs? > > > > > > Ryanne > > > > > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > > > wrote: > > > > > > > Hello everyone, > > > > > > > > We have an interesting feature request from a client regarding having > > the > > > > property of automatic topic creation to be reflected in a MM2. > > Specifically > > > > the current behaviour where if you have automatic topic creation set to > > > > false for the original Kafla cluster, MM2 configuration ignores this > > which > > > > means that if Kafka clients send messages to the MM2 then topics will > > still > > > > be automatically created on target cluster. The core problem here for > > the > > > > client is that our client wants to have complete control over how > > topics > > > > get created (i.e. with terraform setup scripts) and with the current > > > > behaviour this is not possible. > > > > > > > > Of course this poses other problems if we did want to change the > > behaviour > > > > as stated earlier, i.e. if you update the configuration for the > > original > > > > Kafka cluster then you get into open questions about how to reflect > > this > > > > configuration onto the mirror maket (this is why its called "mirror"). > > Is > > > > making MM2 reflect that flag a feature that makes general or > > alternately is > > > > there another variation that makes more sense (i.e. having a separate > > > > specific property rather than reusing the current automatic topic > > creation > > > > one). > > > > > > > > There is a currently existing issue on this at > > > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 what > > are > > > > your thoughts on this? > > > > > > > > > > > > -- > > > > > > > > Matthew de Detrich > > > > > > > > *Aiven Deutschland GmbH* > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > *m:* +491603708037 > > > > > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > > > > > > >
Re: MM2 taking into consideration automatic topic creation property from original cluster
Thanks Ryanne and Mickael I already suspected that something like this (i.e. current behaivior being a deliberate design decision) was the case and I just wanted to confirm my suspicions. I will relay this internally On Fri, 11 Jun 2021, 18:55 Mickael Maison, wrote: > Hi Matthew, > > If an administrator want to control topic creation, they should use > ACLs to prevents users creating topic. Relying on all applications to > not create topics is unlikely to succeed. > > Each refresh.topics.interval.seconds, MM2 checks if it needs to create > topics/partitions by comparing both clusters and also their previous > states. If your automation is able to create new topics on both > clusters, MM2 should detect them correctly and not attempt creating > any topics. If a topic does not exist on the target cluster, MM2 will > try creating it. If it fails, it will retry again at the next > refresh.topics.interval.seconds until the topic gets created. However, > it will also trigger a task reconfiguration each time which may have > an impact on your mirroring throughput. > > On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan > wrote: > > > > Matthew, I wonder what the expected behavior would be when topic-creation > > is disabled and MM is asked to replicate a topic that doesn't exist on > the > > target cluster? Maybe the task should fail at that point, or maybe it > > should replicate whatever it can? > > > > I think the current behavior is reasonable, esp considering precedent > from > > Connect and Streams, both of which actively create topics as needed. > > > > But I understand the motivation. Have they considered revoking topic > > creation permission using ACLs? > > > > Ryanne > > > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > > wrote: > > > > > Hello everyone, > > > > > > We have an interesting feature request from a client regarding having > the > > > property of automatic topic creation to be reflected in a MM2. > Specifically > > > the current behaviour where if you have automatic topic creation set to > > > false for the original Kafla cluster, MM2 configuration ignores this > which > > > means that if Kafka clients send messages to the MM2 then topics will > still > > > be automatically created on target cluster. The core problem here for > the > > > client is that our client wants to have complete control over how > topics > > > get created (i.e. with terraform setup scripts) and with the current > > > behaviour this is not possible. > > > > > > Of course this poses other problems if we did want to change the > behaviour > > > as stated earlier, i.e. if you update the configuration for the > original > > > Kafka cluster then you get into open questions about how to reflect > this > > > configuration onto the mirror maket (this is why its called "mirror"). > Is > > > making MM2 reflect that flag a feature that makes general or > alternately is > > > there another variation that makes more sense (i.e. having a separate > > > specific property rather than reusing the current automatic topic > creation > > > one). > > > > > > There is a currently existing issue on this at > > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 what > are > > > your thoughts on this? > > > > > > > > > -- > > > > > > Matthew de Detrich > > > > > > *Aiven Deutschland GmbH* > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > *m:* +491603708037 > > > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > > > >
Re: MM2 taking into consideration automatic topic creation property from original cluster
Hi Matthew, If an administrator want to control topic creation, they should use ACLs to prevents users creating topic. Relying on all applications to not create topics is unlikely to succeed. Each refresh.topics.interval.seconds, MM2 checks if it needs to create topics/partitions by comparing both clusters and also their previous states. If your automation is able to create new topics on both clusters, MM2 should detect them correctly and not attempt creating any topics. If a topic does not exist on the target cluster, MM2 will try creating it. If it fails, it will retry again at the next refresh.topics.interval.seconds until the topic gets created. However, it will also trigger a task reconfiguration each time which may have an impact on your mirroring throughput. On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan wrote: > > Matthew, I wonder what the expected behavior would be when topic-creation > is disabled and MM is asked to replicate a topic that doesn't exist on the > target cluster? Maybe the task should fail at that point, or maybe it > should replicate whatever it can? > > I think the current behavior is reasonable, esp considering precedent from > Connect and Streams, both of which actively create topics as needed. > > But I understand the motivation. Have they considered revoking topic > creation permission using ACLs? > > Ryanne > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > wrote: > > > Hello everyone, > > > > We have an interesting feature request from a client regarding having the > > property of automatic topic creation to be reflected in a MM2. Specifically > > the current behaviour where if you have automatic topic creation set to > > false for the original Kafla cluster, MM2 configuration ignores this which > > means that if Kafka clients send messages to the MM2 then topics will still > > be automatically created on target cluster. The core problem here for the > > client is that our client wants to have complete control over how topics > > get created (i.e. with terraform setup scripts) and with the current > > behaviour this is not possible. > > > > Of course this poses other problems if we did want to change the behaviour > > as stated earlier, i.e. if you update the configuration for the original > > Kafka cluster then you get into open questions about how to reflect this > > configuration onto the mirror maket (this is why its called "mirror"). Is > > making MM2 reflect that flag a feature that makes general or alternately is > > there another variation that makes more sense (i.e. having a separate > > specific property rather than reusing the current automatic topic creation > > one). > > > > There is a currently existing issue on this at > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 what are > > your thoughts on this? > > > > > > -- > > > > Matthew de Detrich > > > > *Aiven Deutschland GmbH* > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > *m:* +491603708037 > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > >
Re: MM2 taking into consideration automatic topic creation property from original cluster
Matthew, I wonder what the expected behavior would be when topic-creation is disabled and MM is asked to replicate a topic that doesn't exist on the target cluster? Maybe the task should fail at that point, or maybe it should replicate whatever it can? I think the current behavior is reasonable, esp considering precedent from Connect and Streams, both of which actively create topics as needed. But I understand the motivation. Have they considered revoking topic creation permission using ACLs? Ryanne On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich wrote: > Hello everyone, > > We have an interesting feature request from a client regarding having the > property of automatic topic creation to be reflected in a MM2. Specifically > the current behaviour where if you have automatic topic creation set to > false for the original Kafla cluster, MM2 configuration ignores this which > means that if Kafka clients send messages to the MM2 then topics will still > be automatically created on target cluster. The core problem here for the > client is that our client wants to have complete control over how topics > get created (i.e. with terraform setup scripts) and with the current > behaviour this is not possible. > > Of course this poses other problems if we did want to change the behaviour > as stated earlier, i.e. if you update the configuration for the original > Kafka cluster then you get into open questions about how to reflect this > configuration onto the mirror maket (this is why its called "mirror"). Is > making MM2 reflect that flag a feature that makes general or alternately is > there another variation that makes more sense (i.e. having a separate > specific property rather than reusing the current automatic topic creation > one). > > There is a currently existing issue on this at > https://issues.apache.org/jira/browse/KAFKA-12753 > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 what are > your thoughts on this? > > > -- > > Matthew de Detrich > > *Aiven Deutschland GmbH* > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > *m:* +491603708037 > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io >
