RE: how to programatically monitor Kafka availability

[Hohl, Ken]

We're doing this -

every half second
For each broker
get the value of this MBean: 
kafka.server:type=KafkaServer,name=BrokerState
if the value is 4 (this broker is the leader), assume the 
cluster is available

if either couldn't get the MBean from any broker, or none had a value 
of 4, assume the cluster is NOT available

Thanks.

Ken Hohl
Cars.com

-Original Message-
From: hsy...@gmail.com [mailto:hsy...@gmail.com] 
Sent: Thursday, December 17, 2015 1:02 PM
To: users@kafka.apache.org
Subject: Re: how to programatically monitor Kafka availability

Hey Hohl,

I use *partitionsFor
<http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#partitionsFor%28java.lang.String%29>*
method to monitor the partition info for particular topics



On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken <kh...@cars.com> wrote:

> We want to be able to monitor the ability to send messages to Kafka 
> topics.  We want to be aware of the inability to do so before the time 
> we attempt to send a message.  What we're looking for is something 
> like a heartbeat.  The reason we need this is that in our deployment 
> environment, Kafka and its clients will not be co-located.  As such, 
> network issues could cause Kafka to not be available to its client.
>
> We've considered using Zookeeper that's already managing the Kafka 
> cluster but have not been able to determine exactly how we would use it.
>
> We've also considered requesting a JMX MBean periodically and 
> concluding the cluster is not accessible if we can't get the MBean 
> from at least 1 broker.
>
> What is the recommended way of accomplishing what we're trying to do?
>
> Thanks.
>
> Ken Hohl
> Cars.com
>
>

Re: how to programatically monitor Kafka availability

[hsy...@gmail.com]

Hey Hohl,

I use *partitionsFor
*
method to monitor the partition info for particular topics



On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken  wrote:

> We want to be able to monitor the ability to send messages to Kafka
> topics.  We want to be aware of the inability to do so before the time we
> attempt to send a message.  What we're looking for is something like a
> heartbeat.  The reason we need this is that in our deployment environment,
> Kafka and its clients will not be co-located.  As such, network issues
> could cause Kafka to not be available to its client.
>
> We've considered using Zookeeper that's already managing the Kafka cluster
> but have not been able to determine exactly how we would use it.
>
> We've also considered requesting a JMX MBean periodically and concluding
> the cluster is not accessible if we can't get the MBean from at least 1
> broker.
>
> What is the recommended way of accomplishing what we're trying to do?
>
> Thanks.
>
> Ken Hohl
> Cars.com
>
>