Kafka audit logs in Apache Kafka?

Hi, Does Apache Kafka have audit logs equivalent of https://docs.confluent.io/current/security/audit-logs.html or does only Confluent's Kafka have this? Thanks, Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.co

Re: Monitoring Broker/Prod/Cons on Kubernetes

Hi, I missed the original email, but I know Sematext can monitor your containerized/k8sized Kafka. Maybe of interest: https://sematext.com/blog/monitoring-kafka-with-sematext/ Otis On Sun, Sep 29, 2019 at 6:26 AM Pere Urbón Bayes wrote: > Hi, > personally speaking my usual monitoring of ch

Re: Monitoring Mirror Maker Lag & source

Hi, If this lag is exposed via JMX, you can monitor it with https://github.com/sematext/sematext-agent-java You'll just need to adjust the Kafka integration in https://github.com/sematext/sematext-agent-integrations/tree/master/kafka This can ship Kafka metrics to InfluxDB or Sematext. Otis -- M

Re: Kafka monitoring

Hi, Maybe you are looking for something like https://github.com/uber/chaperone ? See also: * https://issues.apache.org/jira/browse/KAFKA-260 * https://sematext.com/blog/2016/06/07/kafka-consumer-lag-offsets-monitoring/ Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Ela

Re: Kafka 0.10 Monitoring tool

Hi, Why are these tools not working perfectly for you? Does it *have to* be open-source? If not, Sematext SPM collects a lot of Kafka metrics, with consumer lag being one of them -- https://sematext.com/blog/2016/06/07/kafka-consumer-lag-offsets-monitoring/ Otis -- Monitoring - Log Management -

Re: Monitoring Kafka client

ng a way to check the > health status programmatically so that I could integrate this into the my > existing monitoring infrastructure. > > Marius > > On Sat, Sep 3, 2016 at 7:23 AM, Otis Gospodnetić < > otis.gospodne...@gmail.com > > wrote: > > > Hi, > > > >

Re: Monitoring Kafka client

Hi, By "kafka client" I assume you mean you Kafka producer and/or consumers? If so, any decent Kafka monitoring solution should let you monitor that. See https://sematext.com/spm/integrations/kafka-monitoring/ for an example. Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr

Re: concept of record vs request vs batch

Hi, Do you control the producers? If so, couldn't you measure the message sizes there? Alternatively, you can use something like SPM for Kafka or other Kafka monitoring tools that expose relevant metrics. For example, I think you can compute avg msg size based on metrics shown in these charts: h

Re: Kafka Monitoring using JMX Mbeans

Hi, On Mon, Apr 25, 2016 at 4:14 AM, Mudit Kumar wrote: > Hi, > > Have anyone setup any monitoring using Mbeans ?What kind of command line > tools been used? > See https://sematext.com/spm/integrations/kafka-monitoring/ We use it for monitoring Kafka, ZooKeeper, Elasticsearch, Solr, etc. Otis

Re: Per Topic Metrics

still, this is not like it says in the documentation. maybe worth > reworking? > > Cheers > > Fabian > > 2015-12-11 21:28 GMT+01:00 Otis Gospodnetić : > > > Hi Fabian, > > > > Hmmm, there are certainly per-topic metrics. > > You can see it in this SPM de

Re: Per Topic Metrics

Hi Fabian, Hmmm, there are certainly per-topic metrics. You can see it in this SPM demo for Kafka monitoring: * click on https://apps.sematext.com/demo to get into the demo account * go into any SPM app with "kafka" in the name, like https://apps.sematext.com/spm-reports/mainPage.do?selectedApplic

Best way to watch for leader changes?

Hi, In Kafka 0.8 Simple Consumer https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example shows how to watch for leader changes and find a new one: if (fetchResponse.hasError()) { // go find new leader // ... Is this still the recommended way of finding new leaders in 0.9?

Re: Consuming "backwards"?

This is an aancient thread, but I thought I'd point to http://blog.sematext.com/2015/11/04/kafka-real-time-stream-multi-topic-catch-up-trick/ which gives a short description of how we ended up implementing this. It seems to work well for us, but if there are better ways to do it, esp. now with

Re: Suggestions on load testing a system that uses kafka as underlying message bus?

Hi, A few pointers are in this Kafka user ML thread: http://search-hadoop.com/m/uyzND177HP92xnm4e Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Sun, Sep 20, 2015 at 1:14 AM, David Luu wrote: > I'd like to

Re: Kafka Server JMX Log End Offset Not updating after repartioning

Replying 3 months later Sounds like http://search-hadoop.com/m/uyzND1YlKxr5XZpK , Joe. No fix yes, as far as I know. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Jun 8, 2015 at 6:33 PM, joe smith

Re: Controlled Shutdown Tool?

Btw. a regular UNIX kill will do the same - SIGTERM - http://linux.die.net/man/1/kill Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Jul 27, 2015 at 3:57 PM, Andrew Otto wrote: > Ah, thank you, SIGTERM

Re: Zombie Replica Fetcher Threads

Juicy one. https://issues.apache.org/jira/browse/KAFKA-2530 I hope it's related to http://search-hadoop.com/m/uyzND1XVyK12UNtd32/kafka+orphaned&subj=Consumer+lag+lies+orphaned+offsets ! :) Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support *

Re: Metrics to monitor in Kafka

Consumer offset/lag is what people are always after. :) See also: http://search-hadoop.com/?q=kafka+otis+important+metrics Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Tue, Aug 25, 2015 at 3:28 AM, Debraj M

Re: Zookeeper jmx monitoring for kafka

Hi Prabhjot, Short answer: yes I used to think ZK was so super stable that it was one of those things that don't require any management, but on a few occasions I witnessed complex distributed applications nearly fall apart because of issues with ZK. We use our own SPM for ZooKeeper to monitor all

Re: How to monitor lag when "kafka" is used as offset.storage?

Hi Shahab - SPM for Kafka captures ~200 Kafka metrics IIRC and has built-in alerting, anomaly detection, and a bunch of other features - see http://sematext.com/spm/integrations/kafka-monitoring.html Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearc

Re: Issue in pulling metrics from kafka-console-producer.

Unfortunately Kafka brokers don't have producer and consumer metrics. :( P & C expose them through their own JMX interfaces. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Sep 10, 2015 at 2:46 AM, Pavan

Re: Consumer lag lies - orphaned offsets?

plication lag is to use the committed > offsets and the current log end offsets. Todd has been doing a lot of > interesting work in making lag monitoring less painful and can comment > more. > > Joel > > On Thu, Jun 04, 2015 at 04:55:44PM -0400, Otis Gospodnetić wrote: > >

Re: Consumer lag lies - orphaned offsets?

; On Thu, Jun 4, 2015 at 4:55 PM, Otis Gospodnetić < > otis.gospodne...@gmail.com > > wrote: > > > Hi, > > > > On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds > > wrote: > > > > > I believe the JMX metrics reflect the consumer PRIOR to committi

Re: Consumer lag lies - orphaned offsets?

Hi, On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds wrote: > I believe the JMX metrics reflect the consumer PRIOR to committing offsets > to Kafka / Zookeeper. But when you query from the command line using the > kafka tools, you are just getting the committed offsets. > Even if that were the ca