Cheng, thanks for the KIP! Can you include some details about how this will work the post-ZK world?
For KafkaAdminClient, will we add a new "internal" field to NewTopic, or will we reuse the existing "configs" map. One concern with sticking this new special field in the topic configs is that we can collide with an existing user-defined "internal" config. Also, what happens if a user tries to alter the config on a topic and changes or removes the "internal" config? If we do not want to separate out "internal" into its own field, I think we'll have to add some guards against users messing with it. It's probably safer to keep it separate. WDYT? -David On Fri, May 29, 2020 at 4:09 AM Cheng Tan <c...@confluent.io> wrote: > Hello developers, > > > I’m proposing KIP-619 to add internal topic creation support. > > Kafka and its upstream applications treat internal topics differently from > non-internal topics. For example: > > • Kafka handles topic creation response errors differently for > internal topics > • Internal topic partitions cannot be added to a transaction > • Internal topic records cannot be deleted > • Appending to internal topics might get rejected > • …… > > Clients and upstream applications may define their own internal topics. > For example, Kafka Connect defines `connect-configs`, `connect-offsets`, > and `connect-statuses`. Clients are fetching the internal topics by sending > the MetadataRequest (ApiKeys.METADATA). > > However, clients and upstream application cannot register their own > internal topics in servers. As a result, servers have no knowledge about > client-defined internal topics. They can only test if a given topic is > internal or not simply by checking against a static set of internal topic > string, which consists of two internal topic names `__consumer_offsets` and > `__transaction_state`. As a result, MetadataRequest cannot provide any > information about client created internal topics. > > To solve this pain point, I'm proposing support for clients to register > and query their own internal topics. > > Please feel free to join the discussion. Thanks in advance. > > > Best, - Cheng Tan -- -David
