Hi Anton, I agree that, while rare, the case you've mentioned where a missing offsets topic leads to reprocessing of potentially massive amounts of data is pretty nasty and it'd be nice to try to prevent it.
Just to make sure we're all on the same page about the problem space here--it seems like the only serious impact here could come from the offsets topic. Creating a new, empty config or status topic should not lead to any problems if the existing one is accidentally deleted. Is this correct? Separately, it seems like the intended flow is a bit awkward: users bring up a worker which automatically creates the internal topics, probably bring up more workers to flesh out the cluster, then have to edit the configs for every worker in the cluster in order to disable internal topic creation. This would be okay in the case of a cluster migration, but it'd be somewhat cumbersome in the general case where a cluster has been brought up and, with no plans of migration, users want to prevent accidental deletion of the offsets topic from causing a ton of data to be reprocessed. An alternative flow could involve introducing a new CLI tool for creating Connect's internal topics, with the intention that it could be used before bringing up any workers in the cluster, and then all workers in the cluster could be configured with internal topic creation disabled from day 1. We could also recommend the use of that tool in the error message if internal topic creation is disabled and a topic isn't found. Thoughts? On 2025/09/12 17:11:12 Anton Liauchuk wrote: > Hi, > > Thanks for the feedback! > > I agree this should be a rare case, but the potential impact could be > catastrophic in situations where many long-running connectors are involved, > as they might start reprocessing everything due to misconfiguration. Such > misconfiguration can happen because of invalid topic names or mistakes > during migration to a new cluster. A recent example of this is the > migration to a new Kafka cluster on AWS, the MSK cluster must be recreated > to migrate from ZooKeeper to KRaft. > > I’ve updated the KIP and PR: > > > - > > Renamed the config to *internal.topics.creation.enable* > - > > Extended the error messages. Example: > > *Topic 'non-existent-offset' specified via the 'offset.storage.topic' > property is missing. The config 'internal.topics.creation.enable' is set to > 'false', so automatic creation of internal topics is disabled. Either > enable automatic creation or create the topics manually before starting the > worker.* > > Please take another look and share your thoughts > > On Fri, Sep 12, 2025 at 6:33 PM Mickael Maison <[email protected]> > wrote: > > > Hi, > > > > Thanks for the KIP. > > > > To be honest I'm unsure about this feature. I find the motivation > > light, this targets a very specific error scenario. > > If we set the new configuration to false, you then must manually > > create the topics before the first start. Also all the internal topic > > configurations (offset.storage.partitions, etc) become unused. > > > > Then regarding the proposed configuration, have you considered > > internal.topics.creation.enable? > > Will the error message also specify the require configurations for the > > missing topics? > > > > Thanks, > > Mickael > > > > On Wed, Sep 10, 2025 at 10:38 AM Andrei Rudkouski > > <[email protected]> wrote: > > > > > > Hi, > > > > > > +1 (non-binding) > > > > > > Thanks for the KIP > > > > > > Best regards, > > > Andrei Rudkouski > > > > > > On 2025/09/09 09:07:14 Anton Liauchuk wrote: > > > > hi > > > > > > > > I’ve added an integration test for this new config: > > > > > > https://github.com/apache/kafka/pull/20384/files#diff-0f86ed224068b85289d9214ae1dad88865d159b8e3658a4231544de36f429bd5R248 > > > > . > > > > > > > > > > > > Please take a look at the KIP - it’s a really small configuration. > > > > > > > > Once we get the required number of votes, the PR can be merged. > > > > > > > > On Thu, Sep 4, 2025 at 9:49 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) > > < > > > > [email protected]> wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > Thanks for the KIP > > > > > > > > > > From: [email protected] At: 09/04/25 13:24:06 UTC-4:00To: > > > > > [email protected] > > > > > Subject: [VOTE] KIP-1209: Add configuration to control internal topic > > > > > creation in Kafka Connect > > > > > > > > > > hi > > > > > > > > > > I would like to start a vote on *KIP-1209: Add configuration to > > control > > > > > internal topic creation in Kafka Connect* > > > > > > > > > > KIP: https://cwiki.apache.org/confluence/x/GAq2Fg > > > > > Discussion thread: > > > > > https://lists.apache.org/thread/l89rzzm0jd4ml7xx66mkxfztst281so7 > > > > > > > > > > > > > > > > > > > > > >
