Gwen, Yes, Chris and I work together; we can put together a KIP.
If someone would please grant me wiki permissions, username is katchison. Thank you. On Mon, Jan 25, 2021 at 2:58 PM Gwen Shapira <g...@confluent.io> wrote: > Agree that this sounds like a good idea. > > Would be good to have a more formal proposal (i.e. a KIP) with the details. > I can think of about 100 different questions (will there be "levels" > like in logs, what type of events are in or out of scope, rate > limiting, data formats, etc). > I am also curious on whether the notifications are intended for > humans, automated processes or even the Kafka client applications > themselves. I hope the proposal can include a few example scenarios to > help us reason about the experience. > > Knowlton, is this something you want to pick up? > > Gwen > > On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon > <christopher.l.shan...@gmail.com> wrote: > > > > Hi, > > > > I am on the ActiveMQ PMC and I think this is a very good idea to have a > way > > to do advisories/notifications/events (whatever you want to call it). In > > ActiveMQ classic you have advisories and in Artemis you have > notifications. > > Having management messages that can be subscribed to in real time is > > actually a major feature that is missing from Kafka that many other > brokers > > have. > > > > The idea here would be to publish notifications of different configurable > > events when something important happens so a consumer can listen in on > > things it cares about and be able to do something instead of having to > poll > > the admin API. There are many events that happen in a broker that would > be > > useful to be notified about. Events such as new connections to the > cluster, > > new topics created or destroyed, consumer group creation, authorization > > errors, new leader election, etc. The list is pretty much endless. > > > > The metadata topic that will exist is probably not going to have all of > > this information so some other mechanism would be needed to handle > > publishing these messages to a specific management topic that would be > > useful for a consumer. > > > > Chris > > > > > > On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen <reluctanthero...@gmail.com> > > wrote: > > > > > Hey Knowles, > > > > > > in Kafka people normally use admin clients to get those metadata. I'm > not > > > sure why you mentioned specifically that having a topic to manage these > > > information is useful, but a good news is that in KIP-500 > > > < > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum > > > > > > > we > > > are trying to deprecate Zookeeper and migrate to a self-managed > metadata > > > topic quorum. At the time this feature is fully done, you should be > able to > > > use consumers to pull the metadata log. > > > > > > Best, > > > Boyang > > > > > > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr < > > > katchiso...@gmail.com> > > > wrote: > > > > > > > Good afternoon all, > > > > > > > > In our Kafka clusters we have a need to know when certain activities > are > > > > performed, mainly topics being created, but brokers coming up/down is > > > also > > > > useful. This would be akin to what ActiveMQ does via advisory > messages ( > > > > https://activemq.apache.org/advisory-message). > > > > > > > > Since there did not appear to be anything in the ecosystem > currently, I > > > > wrote a standalone Java program that watches the various ZooKeeper > > > > locations that the Kafka broker writes to and deltas can tell us > > > > topic/broker actions etc... and writes to a kafka topic for > downstream > > > > consumption. > > > > > > > > Ideally, we would rather have the broker handle this internally > rather > > > > than yet another service stood up in our systems. I began digging > through > > > > the broker source (my Scala is basically hello world level) and there > > > does > > > > not appear to be any mechanism in which this could be easily patched > into > > > > the broker. > > > > > > > > Specifically, a producer or consumer acting upon an nonexistent > topic or > > > a > > > > manual CreateTopic would trigger a Produce to this advisory topic > and the > > > > KafkaApis framework would handle it like any other request. However, > by > > > the > > > > time we are inside the getTopicMetadata call there doesn't seem to > be a > > > > clean way to fire off another message that would make its way through > > > > KafkaApis. Perhaps another XManager type object is required? > > > > > > > > Looking for alternative ideas or guidance (or I missed something in > the > > > > broker). > > > > > > > > Thank you. > > > > > > > > Knowles > > > > > > > > > > > -- > Gwen Shapira > Engineering Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
