HI Otis,
You are right. If the Kafka itself have problem (QUEUE is full, auto
rebalance etc, drop event), how can it transmit the logs... So we have
tried to avoid "agent based" solution Apache Flume Agent or Syslog
configuration.
You are right we have to build a redundant transportation for mo
Hi,
I'm late to the thread... but that "...we intercept log4j..." caught my
attention. Why intercept, especially if it's causing trouble?
Could you use log4j syslog appender and get logs routed to wherever you
want them via syslog, for example?
Or you can have syslog tail log4j log files (e.g. r
Hi All,
Thanks for all your responses.
JMX metrics are there and we do pull the metrics, but I would like to
capture the logs from Kafka lib as well especially WARN, FATAL and ERROR
etc to debug the issue.
To do this, we intercept Log4j logging and send it to Kafka Log Topics, but
I realize
We monitor producers or for that matter any process/service using JMX
metrics. Every server and service in LinkedIn sends metrics in a Kafka
message to a metrics Kafka cluster. We have subscribers that connect to the
metrics cluster to index that data in RRDs.
Our aim is to expose all important me
We use Kafka as Transport Layer to transport application logs. How do we
monitor Producers at large scales about 6000 boxes x 4 topic per box so
roughly 24000 producers (spread across multiple data center.. we have
brokers per DC). We do the monitoring based on logs. I have tried
intercepting lo
