Hi,
I have come across a few stack overflow posts on the subject of request-response type semantics through KAFKA. From some of the approaches that I've read developers are using KAFKA's auto topic create feature (or AdminClient) to dynamically create topics per request-response channel. They mention that the number of processes that wish to communicate can vary, with the processes using some form of id to create unique topics for themselves. See https://stackoverflow.com/questions/35535785/does-kafka-support-request-response-messaging and https://stackoverflow.com/questions/46662102/correlating-in-kafka-and-dynamic-topics/46678198<https://stackoverflow.com/questions/46662102/correlating-in-kafka-and-dynamic-topics/46678198#46678198>. This approach leads to several problems however: 1. The maximum number of such channels (topics) is limited by the memory of a ZK node (in-memory constraints since ZK is not sharded) 2. ZK is best used when reads outnumber writes. Creating topics at high rates could affect ZK cluster performance. 3. Once the communication is done by the initiating process some entity must delete the topic it used since it will never be used again To solve one part of this problem, I find it strange that KAFKA does not provide a Topic Retention Policy (not a log retention policy). This would delete topics which are considered "stale" from the KAFKA cluster and from ZK. By deleting topics for the user, this would reduce the amount of code and administrative headache stale topics currently place on the user. Would this be a feature that the community would find value in, while keeping true to KAFKA's fundamentals and not require substantial refactoring? Alexei
