Hi Robin, I'm interested in a use case in which I need to be able to have a connect cluster fail, and then bring up a new cluster with the same offset topics and connectors. By new cluster I mean a cluster with a new `group.id`. I am aware I could just use the same group id as before but I would like to explore this route.
I'm keen to learn more about the reasons the described case above, and those in my original thread, aren't recommended. Thank you, Jordan On Wed, 30 Mar 2022 at 14:00, Robin Moffatt <ro...@confluent.io.invalid> wrote: > Hi Jordan, > > Is there a good reason for wanting to do this? I can think of multiple > reasons why you shouldn't do this even if technically it works in some > cases. > Or it's just curiosity as to whether you can/should? > > thanks, Robin. > > > -- > > Robin Moffatt | Principal Developer Advocate | ro...@confluent.io | @rmoff > > > On Wed, 30 Mar 2022 at 13:36, Jordan Wyatt <jwyatt1...@gmail.com> wrote: > > > Hi, > > > > I've recently been experimenting with setting the values of the `offset,` > > `storage` and `status` topics within Kafka Connect. > > > > I'm aware from various sources (Robin Moffatt blogs, StackOverflow, > > Confluent Kafka Connect docs) that these topics should not be shared > across > > different connect **clusters**. e.g for each unique set of workers > with a > > given `group.id`, a unique set of internal storage topics should be > used. > > > > These discussions and documentations usually talk about sharing all three > > topics at once, however, I am interested in reusing only the offset > storage > > topic. I am struggling to find the risks of sharing this offset topic > > between different connect clusters. > > > > I'm aware of issues with sharing the config and status topics from blogs > > and my own testing (clusters can end up running connectors from other > > clusters, for example), but I cannot find a case for not sharing the > offset > > topic despite guidance to avoid this. > > > > The use cases I am interested in are: > > > > 1. Sharing an offset topic between clusters, but never in parallel. > > > > > > *e.g cluster 1 running connector A uses the offset topic, cluster 1 and > > connector A are deleted, then cluster 2 running connector B is created > uses > > the offset topic. * > > > > 2. As above, but using the offset topic in parallel. > > > > As the offset.stroage topic is keyed by connector name (from the source > > connectors I've tried) I do not understand the risk of both of the above > > cases **unless** > 1 connector exists with the same name in separate > > clusters, as there would then be the risk of key collision as group.id > is > > not referenced in the offset topic keys. > > > > Any insights into why sharing the offset topic between clusters for the > > cases described would be greatly appreciated, thank you. > > >
