Hi All,

Thanks for all your responses.



JMX metrics are there and we do pull the metrics, but I would like to
capture the logs from Kafka lib as well especially WARN, FATAL and ERROR
etc to debug the issue.



To do this, we intercept Log4j logging and send it to Kafka Log Topics, but
I realize that under heavy Kafka Lib error/warn/  it will create a deadlock
between Producer Send thread  (Logging Kafka log topic queue...)



*public* *class* KafkaLog4jAppender *extends* AppenderSkeleton {



Producer  producer......

*protected* *void* append(LoggingEvent event) {


if(event.getLoggerName().startsWith("kafka")){

     if(event is WARN, FATAL and ERROR){

                        producer.send(event.getRenderedMessage())

            }

}



}


Other option is to log Kafka Logs into disk and transport logs via
separate process
to Kafka Topic and transport via https://github.com/harelba/tail2kafka to
topic...


We use Kafka for Log transportation and we want to debug/trouble shoot
issue via logs or create alerts/etc


Thanks,


Bhavesh




On Wed, Jun 25, 2014 at 10:49 AM, Neha Narkhede <[email protected]>
wrote:

> We monitor producers or for that matter any process/service using JMX
> metrics. Every server and service in LinkedIn sends metrics in a Kafka
> message to a metrics Kafka cluster. We have subscribers that connect to the
> metrics cluster to index that data in RRDs.
>
> Our aim is to expose all important metrics through JMX. We are doing that
> for the new producer under org.apache.kafka.clients.producer. Feel free to
> take a look at that and give feedback.
>
> Thanks,
> Neha
>
>
> On Tue, Jun 24, 2014 at 7:59 PM, Darion Yaphet <[email protected]>
> wrote:
>
> > Sorry I want to  know  you want to monitor kafka producers or kafka
> brokers
> > and zookeepers ?
> > It's seems you will want to monitor monitor Exceptions eg Leader Not
> Found,
> > Queue is full, resend fail  etc  are kafka cluster
> >
> >
> > 2014-06-25 8:20 GMT+08:00 Bhavesh Mistry <[email protected]>:
> >
> > > We use Kafka as Transport Layer to transport application logs.  How do
> we
> > > monitor Producers at large scales about 6000 boxes x 4 topic per box so
> > > roughly 24000 producers (spread across multiple data center.. we have
> > > brokers per DC).  We do the monitoring based on logs.  I have tried
> > > intercepting logs via Log4J custom implementation which only intercept
> > WARN
> > > and ERROR and FATAL events  org.apache.log4j.AppenderSkeleton append
> > method
> > > which send its logs to brokers (This is working but after load testing
> it
> > > is causing deadlock some times between ProducerSendThread and
> Producer).
> > >
> > > I know there are JMX monitoring MBeans available which we can pull the
> > > data, but I would like to monitor Exceptions eg Leader Not Found, Queue
> > is
> > > full, resend fail etc in Kafka Library.
> > >
> > > How does LinkedIn monitor the Producers ?
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> >
> >
> >
> > --
> >
> >
> > long is the way and hard  that out of Hell leads up to light
> >
>

Reply via email to