Hey Kafka Users, When using a Kafka broker w/ auto.create.topics.enable set to true, how do Kafka users generally manage configuration of those topics? In particular, cleanup.policy={compact/delete} can be a crucial configuration value to get correct.
In my application, I have a couple Kafka Connect sources that generate all the input to my Kafka brokers, across about 30 different topics. All of these specific topics are used in a KStreams app as tables, so they should be configured w/ cleanup.policy=compact so that state-store restoration can take place on them without losing data. As far as I know, there's no way for a Kafka producer (or more specifically a KC source) to indicate topic configuration if/when the topic is auto-created, so it needs to be done afterwards. Other topics, such as KStreams repartition topics, should be cleanup.policy=delete, and others such as KStream changelog topics should be cleanup.policy=compact... but KStreams handles all this for me for internal topics. I'm envisioning the ideal solution as something like a scheduled task that runs against my cluster, with a configuration, that identifies and corrects misconfigurations. This could then roll out through different staging and testing areas just like a general software change. But developing such a tool doesn't seem straight-forward since there currently isn't an API for managing topic configuration either... Mathieu