Hey Knowles, in Kafka people normally use admin clients to get those metadata. I'm not sure why you mentioned specifically that having a topic to manage these information is useful, but a good news is that in KIP-500 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum> we are trying to deprecate Zookeeper and migrate to a self-managed metadata topic quorum. At the time this feature is fully done, you should be able to use consumers to pull the metadata log.
Best, Boyang On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <katchiso...@gmail.com> wrote: > Good afternoon all, > > In our Kafka clusters we have a need to know when certain activities are > performed, mainly topics being created, but brokers coming up/down is also > useful. This would be akin to what ActiveMQ does via advisory messages ( > https://activemq.apache.org/advisory-message). > > Since there did not appear to be anything in the ecosystem currently, I > wrote a standalone Java program that watches the various ZooKeeper > locations that the Kafka broker writes to and deltas can tell us > topic/broker actions etc... and writes to a kafka topic for downstream > consumption. > > Ideally, we would rather have the broker handle this internally rather > than yet another service stood up in our systems. I began digging through > the broker source (my Scala is basically hello world level) and there does > not appear to be any mechanism in which this could be easily patched into > the broker. > > Specifically, a producer or consumer acting upon an nonexistent topic or a > manual CreateTopic would trigger a Produce to this advisory topic and the > KafkaApis framework would handle it like any other request. However, by the > time we are inside the getTopicMetadata call there doesn't seem to be a > clean way to fire off another message that would make its way through > KafkaApis. Perhaps another XManager type object is required? > > Looking for alternative ideas or guidance (or I missed something in the > broker). > > Thank you. > > Knowles >
