Agree that this sounds like a good idea. Would be good to have a more formal proposal (i.e. a KIP) with the details. I can think of about 100 different questions (will there be "levels" like in logs, what type of events are in or out of scope, rate limiting, data formats, etc). I am also curious on whether the notifications are intended for humans, automated processes or even the Kafka client applications themselves. I hope the proposal can include a few example scenarios to help us reason about the experience.
Knowlton, is this something you want to pick up? Gwen On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon <christopher.l.shan...@gmail.com> wrote: > > Hi, > > I am on the ActiveMQ PMC and I think this is a very good idea to have a way > to do advisories/notifications/events (whatever you want to call it). In > ActiveMQ classic you have advisories and in Artemis you have notifications. > Having management messages that can be subscribed to in real time is > actually a major feature that is missing from Kafka that many other brokers > have. > > The idea here would be to publish notifications of different configurable > events when something important happens so a consumer can listen in on > things it cares about and be able to do something instead of having to poll > the admin API. There are many events that happen in a broker that would be > useful to be notified about. Events such as new connections to the cluster, > new topics created or destroyed, consumer group creation, authorization > errors, new leader election, etc. The list is pretty much endless. > > The metadata topic that will exist is probably not going to have all of > this information so some other mechanism would be needed to handle > publishing these messages to a specific management topic that would be > useful for a consumer. > > Chris > > > On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen <reluctanthero...@gmail.com> > wrote: > > > Hey Knowles, > > > > in Kafka people normally use admin clients to get those metadata. I'm not > > sure why you mentioned specifically that having a topic to manage these > > information is useful, but a good news is that in KIP-500 > > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum > > > > > we > > are trying to deprecate Zookeeper and migrate to a self-managed metadata > > topic quorum. At the time this feature is fully done, you should be able to > > use consumers to pull the metadata log. > > > > Best, > > Boyang > > > > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr < > > katchiso...@gmail.com> > > wrote: > > > > > Good afternoon all, > > > > > > In our Kafka clusters we have a need to know when certain activities are > > > performed, mainly topics being created, but brokers coming up/down is > > also > > > useful. This would be akin to what ActiveMQ does via advisory messages ( > > > https://activemq.apache.org/advisory-message). > > > > > > Since there did not appear to be anything in the ecosystem currently, I > > > wrote a standalone Java program that watches the various ZooKeeper > > > locations that the Kafka broker writes to and deltas can tell us > > > topic/broker actions etc... and writes to a kafka topic for downstream > > > consumption. > > > > > > Ideally, we would rather have the broker handle this internally rather > > > than yet another service stood up in our systems. I began digging through > > > the broker source (my Scala is basically hello world level) and there > > does > > > not appear to be any mechanism in which this could be easily patched into > > > the broker. > > > > > > Specifically, a producer or consumer acting upon an nonexistent topic or > > a > > > manual CreateTopic would trigger a Produce to this advisory topic and the > > > KafkaApis framework would handle it like any other request. However, by > > the > > > time we are inside the getTopicMetadata call there doesn't seem to be a > > > clean way to fire off another message that would make its way through > > > KafkaApis. Perhaps another XManager type object is required? > > > > > > Looking for alternative ideas or guidance (or I missed something in the > > > broker). > > > > > > Thank you. > > > > > > Knowles > > > > > -- Gwen Shapira Engineering Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
