Re: Kafka Advisory Topic

[Christopher Shannon]

After discussing this a bit more and looking into what it would take to
implement this feature Knowles and I are going to hold off until we see
what the removal of Zookeeper will look like. We are still interested in
this feature but we figure that things will change significantly enough
that it's probably a good idea to wait to see how things change with Kafka
3.0.

On Tue, Jan 26, 2021 at 1:21 PM Knowles Atchison Jr 
wrote:

> Gwen,
>
> Yes, Chris and I work together; we can put together a KIP.
>
> If someone would please grant me wiki permissions, username is katchison.
>
> Thank you.
>
> On Mon, Jan 25, 2021 at 2:58 PM Gwen Shapira  wrote:
>
> > Agree that this sounds like a good idea.
> >
> > Would be good to have a more formal proposal (i.e. a KIP) with the
> details.
> > I can think of about 100 different questions (will there be "levels"
> > like in logs, what type of events are in or out of scope, rate
> > limiting, data formats, etc).
> > I am also curious on whether the notifications are intended for
> > humans, automated processes or even the Kafka client applications
> > themselves. I hope the proposal can include a few example scenarios to
> > help us reason about the experience.
> >
> > Knowlton, is this something you want to pick up?
> >
> > Gwen
> >
> > On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon
> >  wrote:
> > >
> > > Hi,
> > >
> > > I am on the ActiveMQ PMC and I think this is a very good idea to have a
> > way
> > > to do advisories/notifications/events (whatever you want to call it).
> In
> > > ActiveMQ classic you have advisories and in Artemis you have
> > notifications.
> > > Having management messages that can be subscribed to in real time is
> > > actually a major feature that is missing from Kafka that many other
> > brokers
> > > have.
> > >
> > > The idea here would be to publish notifications of different
> configurable
> > > events when something important happens so a consumer can listen in on
> > > things it cares about and be able to do something instead of having to
> > poll
> > > the admin API. There are many events that happen in a broker that would
> > be
> > > useful to be notified about. Events such as new connections to the
> > cluster,
> > > new topics created or destroyed, consumer group creation, authorization
> > > errors, new leader election, etc. The list is pretty much endless.
> > >
> > > The metadata topic that will exist is probably not going to have all of
> > > this information so some other mechanism would be needed to handle
> > > publishing these messages to a specific management topic that would be
> > > useful for a consumer.
> > >
> > > Chris
> > >
> > >
> > > On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Hey Knowles,
> > > >
> > > > in Kafka people normally use admin clients to get those metadata. I'm
> > not
> > > > sure why you mentioned specifically that having a topic to manage
> these
> > > > information is useful, but a good news is that in KIP-500
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > > > >
> > > > we
> > > > are trying to deprecate Zookeeper and migrate to a self-managed
> > metadata
> > > > topic quorum. At the time this feature is fully done, you should be
> > able to
> > > > use consumers to pull the metadata log.
> > > >
> > > > Best,
> > > > Boyang
> > > >
> > > > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> > > > katchiso...@gmail.com>
> > > > wrote:
> > > >
> > > > > Good afternoon all,
> > > > >
> > > > > In our Kafka clusters we have a need to know when certain
> activities
> > are
> > > > > performed, mainly topics being created, but brokers coming up/down
> is
> > > > also
> > > > > useful. This would be akin to what ActiveMQ does via advisory
> > messages (
> > > > > https://activemq.apache.org/advisory-message).
> > > > >
> > > > > Since there did not appear to be anything in the ecosystem
> > currently, I
> > > > > wrote a standalone Java program that watches the various ZooKeeper
> > > > > locations that the Kafka broker writes to and deltas can tell us
> > > > > topic/broker actions etc... and writes to a kafka topic for
> > downstream
> > > > > consumption.
> > > > >
> > > > > Ideally, we would rather have the broker handle this internally
> > rather
> > > > > than yet another service stood up in our systems. I began digging
> > through
> > > > > the broker source (my Scala is basically hello world level) and
> there
> > > > does
> > > > > not appear to be any mechanism in which this could be easily
> patched
> > into
> > > > > the broker.
> > > > >
> > > > > Specifically, a producer or consumer acting upon an nonexistent
> > topic or
> > > > a
> > > > > manual CreateTopic would trigger a Produce to this advisory topic
> > and the
> > > > > KafkaApis framework would handle it like any other request.
> However,
> > by
> > > > the
> > > > > time

Re: Kafka Advisory Topic

[Knowles Atchison Jr]

Gwen,

Yes, Chris and I work together; we can put together a KIP.

If someone would please grant me wiki permissions, username is katchison.

Thank you.

On Mon, Jan 25, 2021 at 2:58 PM Gwen Shapira  wrote:

> Agree that this sounds like a good idea.
>
> Would be good to have a more formal proposal (i.e. a KIP) with the details.
> I can think of about 100 different questions (will there be "levels"
> like in logs, what type of events are in or out of scope, rate
> limiting, data formats, etc).
> I am also curious on whether the notifications are intended for
> humans, automated processes or even the Kafka client applications
> themselves. I hope the proposal can include a few example scenarios to
> help us reason about the experience.
>
> Knowlton, is this something you want to pick up?
>
> Gwen
>
> On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon
>  wrote:
> >
> > Hi,
> >
> > I am on the ActiveMQ PMC and I think this is a very good idea to have a
> way
> > to do advisories/notifications/events (whatever you want to call it). In
> > ActiveMQ classic you have advisories and in Artemis you have
> notifications.
> > Having management messages that can be subscribed to in real time is
> > actually a major feature that is missing from Kafka that many other
> brokers
> > have.
> >
> > The idea here would be to publish notifications of different configurable
> > events when something important happens so a consumer can listen in on
> > things it cares about and be able to do something instead of having to
> poll
> > the admin API. There are many events that happen in a broker that would
> be
> > useful to be notified about. Events such as new connections to the
> cluster,
> > new topics created or destroyed, consumer group creation, authorization
> > errors, new leader election, etc. The list is pretty much endless.
> >
> > The metadata topic that will exist is probably not going to have all of
> > this information so some other mechanism would be needed to handle
> > publishing these messages to a specific management topic that would be
> > useful for a consumer.
> >
> > Chris
> >
> >
> > On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen 
> > wrote:
> >
> > > Hey Knowles,
> > >
> > > in Kafka people normally use admin clients to get those metadata. I'm
> not
> > > sure why you mentioned specifically that having a topic to manage these
> > > information is useful, but a good news is that in KIP-500
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > > >
> > > we
> > > are trying to deprecate Zookeeper and migrate to a self-managed
> metadata
> > > topic quorum. At the time this feature is fully done, you should be
> able to
> > > use consumers to pull the metadata log.
> > >
> > > Best,
> > > Boyang
> > >
> > > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> > > katchiso...@gmail.com>
> > > wrote:
> > >
> > > > Good afternoon all,
> > > >
> > > > In our Kafka clusters we have a need to know when certain activities
> are
> > > > performed, mainly topics being created, but brokers coming up/down is
> > > also
> > > > useful. This would be akin to what ActiveMQ does via advisory
> messages (
> > > > https://activemq.apache.org/advisory-message).
> > > >
> > > > Since there did not appear to be anything in the ecosystem
> currently, I
> > > > wrote a standalone Java program that watches the various ZooKeeper
> > > > locations that the Kafka broker writes to and deltas can tell us
> > > > topic/broker actions etc... and writes to a kafka topic for
> downstream
> > > > consumption.
> > > >
> > > > Ideally, we would rather have the broker handle this internally
> rather
> > > > than yet another service stood up in our systems. I began digging
> through
> > > > the broker source (my Scala is basically hello world level) and there
> > > does
> > > > not appear to be any mechanism in which this could be easily patched
> into
> > > > the broker.
> > > >
> > > > Specifically, a producer or consumer acting upon an nonexistent
> topic or
> > > a
> > > > manual CreateTopic would trigger a Produce to this advisory topic
> and the
> > > > KafkaApis framework would handle it like any other request. However,
> by
> > > the
> > > > time we are inside the getTopicMetadata call there doesn't seem to
> be a
> > > > clean way to fire off another message that would make its way through
> > > > KafkaApis. Perhaps another XManager type object is required?
> > > >
> > > > Looking for alternative ideas or guidance (or I missed something in
> the
> > > > broker).
> > > >
> > > > Thank you.
> > > >
> > > > Knowles
> > > >
> > >
>
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: Kafka Advisory Topic

[Gwen Shapira]

Agree that this sounds like a good idea.

Would be good to have a more formal proposal (i.e. a KIP) with the details.
I can think of about 100 different questions (will there be "levels"
like in logs, what type of events are in or out of scope, rate
limiting, data formats, etc).
I am also curious on whether the notifications are intended for
humans, automated processes or even the Kafka client applications
themselves. I hope the proposal can include a few example scenarios to
help us reason about the experience.

Knowlton, is this something you want to pick up?

Gwen

On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon
 wrote:
>
> Hi,
>
> I am on the ActiveMQ PMC and I think this is a very good idea to have a way
> to do advisories/notifications/events (whatever you want to call it). In
> ActiveMQ classic you have advisories and in Artemis you have notifications.
> Having management messages that can be subscribed to in real time is
> actually a major feature that is missing from Kafka that many other brokers
> have.
>
> The idea here would be to publish notifications of different configurable
> events when something important happens so a consumer can listen in on
> things it cares about and be able to do something instead of having to poll
> the admin API. There are many events that happen in a broker that would be
> useful to be notified about. Events such as new connections to the cluster,
> new topics created or destroyed, consumer group creation, authorization
> errors, new leader election, etc. The list is pretty much endless.
>
> The metadata topic that will exist is probably not going to have all of
> this information so some other mechanism would be needed to handle
> publishing these messages to a specific management topic that would be
> useful for a consumer.
>
> Chris
>
>
> On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen 
> wrote:
>
> > Hey Knowles,
> >
> > in Kafka people normally use admin clients to get those metadata. I'm not
> > sure why you mentioned specifically that having a topic to manage these
> > information is useful, but a good news is that in KIP-500
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > >
> > we
> > are trying to deprecate Zookeeper and migrate to a self-managed metadata
> > topic quorum. At the time this feature is fully done, you should be able to
> > use consumers to pull the metadata log.
> >
> > Best,
> > Boyang
> >
> > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> > katchiso...@gmail.com>
> > wrote:
> >
> > > Good afternoon all,
> > >
> > > In our Kafka clusters we have a need to know when certain activities are
> > > performed, mainly topics being created, but brokers coming up/down is
> > also
> > > useful. This would be akin to what ActiveMQ does via advisory messages (
> > > https://activemq.apache.org/advisory-message).
> > >
> > > Since there did not appear to be anything in the ecosystem currently, I
> > > wrote a standalone Java program that watches the various ZooKeeper
> > > locations that the Kafka broker writes to and deltas can tell us
> > > topic/broker actions etc... and writes to a kafka topic for downstream
> > > consumption.
> > >
> > > Ideally, we would rather have the broker handle this internally rather
> > > than yet another service stood up in our systems. I began digging through
> > > the broker source (my Scala is basically hello world level) and there
> > does
> > > not appear to be any mechanism in which this could be easily patched into
> > > the broker.
> > >
> > > Specifically, a producer or consumer acting upon an nonexistent topic or
> > a
> > > manual CreateTopic would trigger a Produce to this advisory topic and the
> > > KafkaApis framework would handle it like any other request. However, by
> > the
> > > time we are inside the getTopicMetadata call there doesn't seem to be a
> > > clean way to fire off another message that would make its way through
> > > KafkaApis. Perhaps another XManager type object is required?
> > >
> > > Looking for alternative ideas or guidance (or I missed something in the
> > > broker).
> > >
> > > Thank you.
> > >
> > > Knowles
> > >
> >



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Kafka Advisory Topic

[Christopher Shannon]

Hi,

I am on the ActiveMQ PMC and I think this is a very good idea to have a way
to do advisories/notifications/events (whatever you want to call it). In
ActiveMQ classic you have advisories and in Artemis you have notifications.
Having management messages that can be subscribed to in real time is
actually a major feature that is missing from Kafka that many other brokers
have.

The idea here would be to publish notifications of different configurable
events when something important happens so a consumer can listen in on
things it cares about and be able to do something instead of having to poll
the admin API. There are many events that happen in a broker that would be
useful to be notified about. Events such as new connections to the cluster,
new topics created or destroyed, consumer group creation, authorization
errors, new leader election, etc. The list is pretty much endless.

The metadata topic that will exist is probably not going to have all of
this information so some other mechanism would be needed to handle
publishing these messages to a specific management topic that would be
useful for a consumer.

Chris


On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen 
wrote:

> Hey Knowles,
>
> in Kafka people normally use admin clients to get those metadata. I'm not
> sure why you mentioned specifically that having a topic to manage these
> information is useful, but a good news is that in KIP-500
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> >
> we
> are trying to deprecate Zookeeper and migrate to a self-managed metadata
> topic quorum. At the time this feature is fully done, you should be able to
> use consumers to pull the metadata log.
>
> Best,
> Boyang
>
> On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> katchiso...@gmail.com>
> wrote:
>
> > Good afternoon all,
> >
> > In our Kafka clusters we have a need to know when certain activities are
> > performed, mainly topics being created, but brokers coming up/down is
> also
> > useful. This would be akin to what ActiveMQ does via advisory messages (
> > https://activemq.apache.org/advisory-message).
> >
> > Since there did not appear to be anything in the ecosystem currently, I
> > wrote a standalone Java program that watches the various ZooKeeper
> > locations that the Kafka broker writes to and deltas can tell us
> > topic/broker actions etc... and writes to a kafka topic for downstream
> > consumption.
> >
> > Ideally, we would rather have the broker handle this internally rather
> > than yet another service stood up in our systems. I began digging through
> > the broker source (my Scala is basically hello world level) and there
> does
> > not appear to be any mechanism in which this could be easily patched into
> > the broker.
> >
> > Specifically, a producer or consumer acting upon an nonexistent topic or
> a
> > manual CreateTopic would trigger a Produce to this advisory topic and the
> > KafkaApis framework would handle it like any other request. However, by
> the
> > time we are inside the getTopicMetadata call there doesn't seem to be a
> > clean way to fire off another message that would make its way through
> > KafkaApis. Perhaps another XManager type object is required?
> >
> > Looking for alternative ideas or guidance (or I missed something in the
> > broker).
> >
> > Thank you.
> >
> > Knowles
> >
>

Re: Kafka Advisory Topic

[Boyang Chen]

Hey Knowles,

in Kafka people normally use admin clients to get those metadata. I'm not
sure why you mentioned specifically that having a topic to manage these
information is useful, but a good news is that in KIP-500

we
are trying to deprecate Zookeeper and migrate to a self-managed metadata
topic quorum. At the time this feature is fully done, you should be able to
use consumers to pull the metadata log.

Best,
Boyang

On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr 
wrote:

> Good afternoon all,
>
> In our Kafka clusters we have a need to know when certain activities are
> performed, mainly topics being created, but brokers coming up/down is also
> useful. This would be akin to what ActiveMQ does via advisory messages (
> https://activemq.apache.org/advisory-message).
>
> Since there did not appear to be anything in the ecosystem currently, I
> wrote a standalone Java program that watches the various ZooKeeper
> locations that the Kafka broker writes to and deltas can tell us
> topic/broker actions etc... and writes to a kafka topic for downstream
> consumption.
>
> Ideally, we would rather have the broker handle this internally rather
> than yet another service stood up in our systems. I began digging through
> the broker source (my Scala is basically hello world level) and there does
> not appear to be any mechanism in which this could be easily patched into
> the broker.
>
> Specifically, a producer or consumer acting upon an nonexistent topic or a
> manual CreateTopic would trigger a Produce to this advisory topic and the
> KafkaApis framework would handle it like any other request. However, by the
> time we are inside the getTopicMetadata call there doesn't seem to be a
> clean way to fire off another message that would make its way through
> KafkaApis. Perhaps another XManager type object is required?
>
> Looking for alternative ideas or guidance (or I missed something in the
> broker).
>
> Thank you.
>
> Knowles
>

Kafka Advisory Topic

[Knowles Atchison Jr]

Good afternoon all,

In our Kafka clusters we have a need to know when certain activities are
performed, mainly topics being created, but brokers coming up/down is also
useful. This would be akin to what ActiveMQ does via advisory messages (
https://activemq.apache.org/advisory-message).

Since there did not appear to be anything in the ecosystem currently, I
wrote a standalone Java program that watches the various ZooKeeper
locations that the Kafka broker writes to and deltas can tell us
topic/broker actions etc... and writes to a kafka topic for downstream
consumption.

Ideally, we would rather have the broker handle this internally rather
than yet another service stood up in our systems. I began digging through
the broker source (my Scala is basically hello world level) and there does
not appear to be any mechanism in which this could be easily patched into
the broker.

Specifically, a producer or consumer acting upon an nonexistent topic or a
manual CreateTopic would trigger a Produce to this advisory topic and the
KafkaApis framework would handle it like any other request. However, by the
time we are inside the getTopicMetadata call there doesn't seem to be a
clean way to fire off another message that would make its way through
KafkaApis. Perhaps another XManager type object is required?

Looking for alternative ideas or guidance (or I missed something in the
broker).

Thank you.

Knowles