Hello developers,
I’m proposing KIP-619 to add internal topic creation support.
Kafka and its upstream applications treat internal topics differently from
non-internal topics. For example:
• Kafka handles topic creation response errors differently for internal
topics
• Internal topic partitions cannot be added to a transaction
• Internal topic records cannot be deleted
• Appending to internal topics might get rejected
• ……
Clients and upstream applications may define their own internal topics. For
example, Kafka Connect defines `connect-configs`, `connect-offsets`, and
`connect-statuses`. Clients are fetching the internal topics by sending the
MetadataRequest (ApiKeys.METADATA).
However, clients and upstream application cannot register their own internal
topics in servers. As a result, servers have no knowledge about client-defined
internal topics. They can only test if a given topic is internal or not simply
by checking against a static set of internal topic string, which consists of
two internal topic names `__consumer_offsets` and `__transaction_state`. As a
result, MetadataRequest cannot provide any information about client created
internal topics.
To solve this pain point, I'm proposing support for clients to register and
query their own internal topics.
Please feel free to join the discussion. Thanks in advance.
Best, - Cheng Tan
