Re: Tools to monitor kafka

SPM for Kafka http://sematext.com/spm/integrations/kafka-monitoring.html Otis > On Nov 20, 2015, at 13:32, Mohit Anchlia wrote: > > Are there any command line or UI tools available to monitor kafka?

Re: High CPU usage for idle kafka server

How's their garbage collection doing? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Jun 4, 2015 at 12:43 AM, Anuja Pundlik (apundlik) < apund...@cisco.com> wrote: > Hi, > > I am using Kafka 0.8.2.1. > W

Re: Consumer lag lies - orphaned offsets?

r_topic-new-cdh,partition=25 Value java.lang.Object = 0 Somebody's lying here. :) Any ideas? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Jun 4, 2015 at 3:54 PM, Otis Gospodn

Re: Consumer lag lies - orphaned offsets?

== could this be a Kafka metrics/JMX bug? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ > > On Tue, Jun 2, 2015 at 4:00 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com > > wrote:

Re: Query on broker info

Hi Pavan, See Monitoring section at https://kafka.apache.org/08/ops.html for various JMX Beans that will tell you about Broker, Producer, and Consumer. See http://sematext.com/spm/integrations/kafka-monitoring.html if you are on 0.8.2 and need something quickly. Otis -- Monitoring * Alerting * A

Consumer lag lies - orphaned offsets?

Hi, I've noticed that when we restart our Kafka consumers our consumer lag metric sometimes looks "weird". Here's an example: https://apps.sematext.com/spm-reports/s/0Hq5zNb4hH You can see lag go up around 15:00, when some consumers were restarted. The "weird" thing is that the lag remains flat!

Re: Offset management: client vs broker side responsibility

Hi, I haven't followed the changes to offset tracking closely, other than that storing them in ZK is not the only option any more. I think what Stevo is asking about/suggesting is that there there be a single API from which offset information can be retrieved (e.g. by monitoring tools), so that mo

Re: Kafka JMS metrics meaning

Hi, On Tue, Jun 2, 2015 at 12:50 PM, Marina wrote: > Hi, > I have enabled JMX_PORT for KAfka server and am trying to understand some > of the metrics that are being exposed. I have two questions: > 1. what are the best metrics to monitor to quickly spot unhealthy Kafka > cluster? > People l

Re: Fw: How to measure performance of Mirror Maker

Hi, I don't know much about MM yet, but what sort of MM performance are you looking for? If MM acts as just another producer for another Kafka cluster (just a guess!) then you could look at the metrics of the consuming cluster to see how fast it's ingesting but if MM works differently, then

Re: Kafka Monitoring using JMX

Hi, Not sure what you are using to get the data out, but if you use SPM you don't have to worry about any of that... assuming you use either 0.7.x or 0.8.2.x. Kafka metrics have changed drastically in the past and it looks like more chan

Re: Kafka start server throws could not reserve enough space for object heap

Short version: probably don't have enough free memory relative to what you have in -Xmx. top, free -m, and other monitoring tools and utilities will share more light on this. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://semate

Re: Consumer Group Lag Reporting

Hi, We track our Consumer Lag and all other Kafka metrics with SPM whose agent collects this info from Kafka Consumers. Here's a chart of that: https://apps.sematext.com/spm-reports/s/mcBHvsXwgj -- this public chart doesn't show all filtering option, but you can filter l

Re: Monitoring of consumer group lag

Mathias, SPM for Kafka will give you Consumer Offsets by Host, Consumer Id, Topic, and Partition, and you can alert (thresholds and/or anomalies) on any combination of these, and of course on any of the other 100+ Kafka metrics there. See http://blog.sematext.com/2015/02/10/kafka-0-8-2-monitoring/

Re: Alternative to camus

Just curious - why - is Camus not suitable/working? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Fri, Mar 13, 2015 at 2:33 PM, Alberto Miorin wrote: > I was wondering if anybody has already tried t

Re: Kafka Mailing List for General Questions

Looks like you subscribed. Just start a new thread and ask away. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Mar 9, 2015 at 4:27 PM, Mark Flores wrote: > Hi, > > > > I would like to subscribe to th

Re: Possible to count for unclosed resources in process

Hi, I think you can look at open file descriptors (network connections use FDs). For example: https://apps.sematext.com/spm-reports/s/IoQDvdT0Ig -- all good https://apps.sematext.com/spm-reports/s/v5Hvwta7PP -- Otis restarting 2 consumers lsof probably shows it, too. Otis -- Monitoring * Alert

Re: Increasing the throughput of Kafka Publisher

Roger, Consider using rsyslog with omkafka. rsyslog rocks! And it's pretty popular, too - http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/ Oh, and it's FAST - some numbers and charts with an older version from 1 year ago: http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasti

Re: Best way to show lag?

Hi, On Sat, Feb 28, 2015 at 9:16 AM, Gene Robichaux wrote: > What is the best way to detect consumer lag? > > We are running each consumer as a separate group and I am running the > ConsumerOffsetChecker to assess the partitions and the lag for each > group/consumer. I run this every 5 minutes.

Re: If you run Kafka in AWS or Docker, how do you persist data?

Hi, On Fri, Feb 27, 2015 at 1:36 AM, James Cheng wrote: > Hi, > > I know that Netflix might be talking about "Kafka on AWS" at the March > meetup, but I wanted to bring up the topic anyway. > > I'm sure that some people are running Kafka in AWS. I'd say most, not some :) > Is anyone running

Re: Kafka Poll: Version You Use?

gt; Christian > > On Wed, Mar 4, 2015 at 8:08 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com > > wrote: > > > Hello hello, > > > > Results of the poll are here! > > Any guesses before looking? > > What % of Kafka users are on 0.8.2.x alre

Re: Kafka Poll: Version You Use?

Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Feb 26, 2015 at 3:32 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > With 0.8.2 out I thought it might be useful for everyone to see which > version(s) of Kafka people are using. &

Re: Kafka metrics: high level vs simple consumer

Bok Stevo, Simple as well, if I'm not mistaken. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Fri, Feb 27, 2015 at 2:35 AM, Stevo Slavić wrote: > Hello Apache Kafka community, > > In Kafka 0.8.1.1, are Kaf

Kafka Poll: Version You Use?

Hi, With 0.8.2 out I thought it might be useful for everyone to see which version(s) of Kafka people are using. Here's a quick poll: http://blog.sematext.com/2015/02/23/kafka-poll-version-you-use/ We'll publish the results next week. Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * C

Re: How to measure performance metrics

Have a look at http://blog.sematext.com/2015/02/10/kafka-0-8-2-monitoring/ There are also various open-source projects. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Wed, Feb 25, 2015 at 12:40 AM, Bhuvana Bas

Re: Broker w/ high memory due to index file sizes

Hi, I'd start small and grow if needed in general. There is one bit of info missing here. Did you try with a smaller heap and had issues? If so, which issues? A few references from our internal Wiki: - XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:

Re: Kafka on Mesos?

Hi Jeff, I see a patchless JIRA issue via http://search-hadoop.com/?q=%2Bmesos+%2Bkafka And if Kafka on YARN is of interest, there is KOYA: http://search-hadoop.com/?q=koya Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://semate

Re: Ship Kafka in on prem product

epository > > which includes RPM and dev packages > > > > http://repository.stratio.com/sds/1.1/ubuntu/13.10/binary/ > > > > http://repository.stratio.com/sds/1.1/RHEL/ > > > > > > > > Regards. > > > > > > 2015-02-11 19:22 GMT+

Re: Ship Kafka in on prem product

Hi, For what it's worth, we include Kafka (now 0.8.2) in our On Premises version of SPM and Logsene. Kafka is used to to transport metrics and logs there. I do seem to recall some not-so-elegant stuff we had to do around packaging, though. I think because http://kafka.apache.org/downloads.ht

ANNOUNCE: Kafka 0.8.2 Monitoring in SPM

Hi, I'm super happy to announce we've added Kafka 0.8.2 monitoring support to SPM. It looks like we capture over 100 Kafka metrics (yes, including that Consumer Lag everyone always wants to see!). Details + charts in: http://blog.sematext.com/2015/02/10/kafka-0-8-2-monitoring/ Thanks to the Ka

Re: Not found NewShinyProducer sync performance metrics

Not announced yet, but http://sematext.com/spm should be showing you all the new shiny Kafka (new producer) metrics out of the box. If you don't see them, please shout (I know we have a bit more tweaking to do in the coming day-two-three). If you want to just dump MBeans from JMX manually and eye

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2.0 Released

Big thanks to Jun and everyone else involved! We're on 0.8.2 as of today. :) Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Tue, Feb 3, 2015 at 8:37 PM, Jun Rao wrote: > The Apache Kafka community is please

Re: kafka out-of-box monitoring system

Hi, Not sure if you are asking if anyone has bundled all these together or if you are asking for an alternative. If the latter, see SPM for Kafka (works with 0.8.2 metrics) (or some other more complete monitoring tool) which will give you everything but 5. (but 5. is in

Re: Logstash to Kafka

Hi, In short, I don't see Kafka having problems with those numbers. Logstash will have a harder time, I believe. That said, it all depends on how you tune things an what kind of / how much hardware you use. 2B or 200B events, yes, big numbers, but how quickly do you need to process those? in 1 m

Re: New Producer - ONLY sync mode?

Can Y have a callback that will handle the notification to X? > In this case, perhaps Y can be async and X can buffer the data until > the callback triggers and says "all good" (or resend if the callback > indicates an error) > > On Mon, Feb 2, 2015 at 12:56 PM, Otis Gospodn

Re: New Producer - ONLY sync mode?

ou need to set > > the batch size to 1 (in producer config) and wait on the future you > > get from Send (i.e. future.get) > > > > I can't think of good reasons to do so, though. > > > > Gwen > > > > > > On Mon, Feb 2, 2015 at 11:08 AM, Oti

Re: Any Java 7 compatibility issues for 0.8.1.1?

I don't think there are any issues. We use 0.8.1.1 under Oracle Java 7. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Feb 2, 2015 at 5:02 AM, Yury Ruchin wrote: > Hello, > > I wonder if there are any

New Producer - ONLY sync mode?

Hi, Is the plan for New Producer to have ONLY async mode? I'm asking because of this info from the Wiki: - The producer will always attempt to batch data and will always immediately return a SendResponse which acts as a Future to allow the client to await the completion of the request.

Re: Resilient Producer

Fernando, have a look - http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/ Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Wed, Jan 28, 2015 at 1:39 PM, Fernando O. wrote: > Hi all, > I'

Re: Poll: Producer/Consumer impl/language you use?

> > > > > > Kafka is included in the Spark examples and external modules and is > > popular > > > as a component of ecosystems on Spark (for which scala is the primary > > > language). > > > > > > 2015-01-28 8:51 GMT-08:00 Otis Gospodne

Poll RESULTS: Producer/Consumer languages

Hi, I promised to share the results of this poll, and here they are: http://blog.sematext.com/2015/01/28/kafka-poll-results-producer-consumer/ List of "surprises" is there. I wonder if anyone else is surprised by any aspect of the breakdown, or is the breakdown just as you expected? Otis -- Mo

Re: Poll: Producer/Consumer impl/language you use?

l go away unfortunately) > > On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi, > > > > I was wondering which implementations/languages people use for their > Kafka > > Producer/Consumers not

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

Hi, Don't use Graphite, so I don't know. Kyle, maybe you can share more info? What do you mean by "reported to Yammer" for example? And when you say Yammer/Graphite, are you trying to say that you are using the Graphite Reporter? If so, can you try other Yammer Reporters and see if there is a m

Poll: Producer/Consumer impl/language you use?

Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have en

Re: Backups

Could one use ZFS or BTRFS snapshot functionality for this? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 1:39 AM, Gwen Shapira wrote: > Hi, > > As a former DBA, I hear you on backups :

Re: dumping JMX data

Hi Scott, http://sematext.com/spm is one of the services Joe was referring to. Jmxc is a command line tool that will dump JMX, see Sematext github account. Otis > On Jan 16, 2015, at 21:00, Scott Chapman wrote: > > Thanks, I actually ran into those already. I was hoping just to be able to

Re: KOYA vs. Samza?

he messages, such > as fault tolerance (restarting consumers/producers when they fail), > checkpointing (saving offsets), state management (if you're counting > messages, you want to make sure your count is accurate even if you fail), > etc. > > Cheers, > Chris > &

KOYA vs. Samza?

Hi, I was wondering if anyone can compare and contrast KOYA and Samza? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/

Re: Monitor data to missing topic

I don't remember seeing a metric for that but I'd look at logs first. Not sure if anything is logged about it, but that's where I'd look for errors. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Jan

Re: Kafka Cluster Monitoring and Documentation of Internals (JMX Metrics) of Rejected Events

Hi, I think you could just email user@? There was no attached image. I think Jun committed something about this: https://issues.apache.org/jira/browse/KAFKA-1481?focusedCommentId=14272057&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14272057 Otis -- Monitoring * A

Re: new to kafka

Hi, Check http://search-hadoop.com/?q=benchmark&fc_project=Kafka for info about Kafka benchmarking. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Sun, Jan 11, 2015 at 8:03 PM, Alpesh Desai wrote: > I am fa

Re: no space left error

Hi, Your disk is full. You should probably have something that checks/monitors disk space and alerts you when it's full. Maybe you can point Kafka to a different, larger disk or partition. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Suppor

Re: Release Date for Kafka 0.8.2

Hi Srividhya, See http://search-hadoop.com/m/4TaT4B9tys1/&subj=Re+Kafka+0+8+2+release+before+Santa+Claus Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Jan 5, 2015 at 11:55 AM, Srividhya Shanmugam < sriv

Re: Latency Tracking Across All Kafka Component

Hi, That sounds a bit like needing a full, cross-app, cross-network transaction/call tracing, and not something specific or limited to Kafka, doesn't it? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Ja

Re: Kafka 0.8.2 release - before Santa Claus?

%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%200.8.2 > > > > > > So, we likely will have another 0.8.2 release in Dec. However, I am not > > > sure if t

Re: kafka monitoring system

Hi Sa Li, Have a look at SPM for monitoring Kafka: http://sematext.com/spm http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/ https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+FAQ#SPMFAQ-KafkaMonitoring Otis -- Monitoring * Alerting * Anomaly Detection * C

Re: kafka-producer-perf-test.sh

If that kafka-producer-per-test.sh doesn't work for you and you choose the DIY approach, https://github.com/sematext/ActionGenerator may be of help. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Dec 22,

Re: How to raise a question in forum

; Shouldn't be. Not sure what the meaning of that would be? You see a negative value somewhere? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ > On Tue, Dec 9, 2014 at 5:23 PM, Otis Gospodnet

Re: How to raise a question in forum

Nitin, If you don't want to DIY there are already tools and solutions for monitoring Kafka, including consumer lag. Ours is mentioned here: http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/ Others have been mentioned on the list a few times before: http://sea

Re: Re: How to push metrics to graphite - jmxtrans does not work

Hi, You can make use of this documentation aimed at JMX and monitoring: https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone There is a section about Kafka and the information is not SPM-specific. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Manageme

Re: maximum number of open file handles

This is not an absolute rule/number. It depends on your Kafka usage and your existing limits. SPM will show you the number of open files Kafka is using, as will a tool like 'lsof'. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & El

Kafka 0.8.2 release - before Santa Claus?

Hi, People using SPM to monitor Kafka have been anxiously asking us about the 0.8.2 release and we've been telling them December. Is that still the plan? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/

Re: Aeron, a high throughput, low latency messaging system

Here's a good writeup that also mentions Kafka: http://highscalability.com/blog/2014/11/17/aeron-do-we-really-need-another-messaging-system.html Comparing it to Kafka is an apples to oranges comparison from what I can tell. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Mana

Re: Getting Simple consumer details using MBean

Hi Madhukar, Maybe you want to look at SPM , which has Kafka monitoring. But please note this: https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+FAQ#SPMFAQ-WhyamInotseeingallKafkametricsifI'mrunninga0.8.xversionofKafkathatispre-0.8.2? Otis -- Monitoring * Alerting *

Re: Programmatic Kafka version detection/extraction?

t;>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> So it looks like we can use Gradle to add properties to > > > >> manifest > > > >>>>> file > > > >>>>>>>

Re: Programmatic Kafka version detection/extraction?

avesh > >> >>> > >> >>> On Tue, Nov 11, 2014 at 10:34 AM, Gwen Shapira < > gshap...@cloudera.com > >> > > >> >>> wrote: > >> >>> > >> >>>> In Sqoop we do the following: > >> >

Re: Programmatic Kafka version detection/extraction?

t; Otis, > > We don't have an api for that now. We can probably expose this as a JMX as > part of kafka-1481. > > Thanks, > > Jun > > On Mon, Nov 10, 2014 at 7:17 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi, > > >

Programmatic Kafka version detection/extraction?

Hi, Is there a way to detect which version of Kafka one is running? Is there an API for that, or a constant with this value, or maybe an MBean or some other way to get to this info? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support

Dynamically adding Kafka brokers

Hi, How do people handle situations, and specifically the broker.id property, where the Kafka (broker) cluster is not fully defined right away? Here's the use case we have at Sematext: * Our software ships as a VM * All components run in this single VM, including 1 Kafka broker * Of course, this

Re: MBeans, dashes, underscores, and KAFKA-1481

gt; > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede < > > neha.narkh...@gmail.com> > > > > wrote: > > > > > > > >> I am

Re: Monitoring connection with kafka client

Hi, We use our own SPM to monitor our Kafka brokers, producers, and consumers (and ZK) and have alerts and anomaly detection on several key Kafka and ZK metrics. When things break around ZK and/or Kafka. we find out pretty quickly because a lot of metrics suddenly s

0.8.x => 0.8.2 upgrade - live & seamless?

Hi, Some of our SPM users who are eager to monitor their Kafka 0.8.x clusters with SPM are asking us whether the upgrade to 0.8.2 from 0.8.1 will be seamless. I believe this will be the case, but wanted to double-check on that... Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centra

Re: MBeans, dashes, underscores, and KAFKA-1481

guys can help provide a patch in a few days, we can probably > get it to the 0.8.2 release. > > Thanks, > > Jun > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi Jun, > > > > I think your MBean renami

Re: MBeans, dashes, underscores, and KAFKA-1481

have to know > which part is the clientid and which part is the topic). > > Thanks, > > Jun > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi Jun, > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun

Faster segment recovery after reboot - 0.8.1

Hi, Just got a lovely email a bunch of our EC2 instances will be rebooted in a few days. Some of them run our Kafka 0.8.1 brokers with a few hundred GBs of data in them. Last time Kafka brokers didn't shut down cleanly it took them mny hours to recover. I just found https://issues.apache.or

Re: Disactivating Yammer Metrics Monitoring

Hi, I don't have any source or configs handy to check things, but you are saying you've configured Kafka to use GraphiteReporter, right? So why not remove that config, so metrics stop being sent to Graphite if your Graphite setup is suffering? If you do that and you still want to see your Kafka

Re: MBeans, dashes, underscores, and KAFKA-1481

s already( > > ".","-","|" etc) in previous version of producer/consumer of Kafka, what > > happens after the upgrade new producer or consumer (kafka version) ? > Also, > > in new producer API (Kafka Trunk), does this enforce the rule abo

Re: MBeans, dashes, underscores, and KAFKA-1481

d Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > *Problem:* > Some Kafka 0.8.x MBeans have names composed of things like group>--. Note how dashes are us

MBeans, dashes, underscores, and KAFKA-1481

Hi, *Problem:* Some Kafka 0.8.x MBeans have names composed of things like --. Note how dashes are used as delimiters. When and don't contain the delimiter character all is good if you want to extract parts of this MBean name by simply splitting on the delimiter character. The problem is that

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi Bhavesh, Yes, if you look at https://issues.apache.org/jira/browse/KAFKA-1481 it describes this sort of a problem with various delimiters used by Kafka. As Jun suggested, I'll start a new thread and try to clearly explain the problem. Otis -- Monitoring * Alerting * Anomaly Detection * Central

Re: GenericJMX plugin file for Kafka 0.8.1

Hi, There's also SPM which makes use of Kafka JMX and does graphing, alerting, and anomaly detection on, I believe, all Kafka metrics for 0.7.x and 0.8.x. You can see a couple Kafka clusters at https://apps.sematext.com/demo . Otis -- Monitoring * Alerting * Anomaly De

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi, > I have topic name with “.” So, it is hard to distinguish metric name and topic If your topics or hosts have dashes or underscores or your topic has dots you will have trouble extracting things from JMX. We provided a patch to change delimiter character in MBeans and are hoping it gets inc

Re: Support for Java 1.8?

Just curious if you saw any issues with Java 1.8 or if everything went smoothly? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Jun 17, 2014 at 3:28 PM, Todd Palino wrote: > We¹re about to start testing in the next

Re: Is there a way to stop/pause producers from writing?

Hi, Wouldn't you want to tell your app to stop producing messages then? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Aug 5, 2014 at 7:45 PM, Stephen Chan wrote: > Is there a tool or way using the Kafka admin api

Re: Issues with metrics collection

Hi, May also want to share version of Kafka. We have SPM agents for Kafka 0.7.x and 0.8.x and we haven't seen that. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Jul 30, 2014 at 10:12 PM, Kiran Nagasubramanian wr

Re: Performance/Stress tools

Would this work for Kafka, Steve, considering Kafka doesn't use Yarn? Dayo - we developed https://github.com/sematext/ActionGenerator to feed events to things like Elasticsearch, Solr, and MongoDB. We'd gladly take a pull request for Kafka. SPM for Kafka

Re: Monitoring Producers at Large Scale

Hi, I'm late to the thread... but that "...we intercept log4j..." caught my attention. Why intercept, especially if it's causing trouble? Could you use log4j syslog appender and get logs routed to wherever you want them via syslog, for example? Or you can have syslog tail log4j log files (e.g. r

Re: Largest known Kafka deployment?

Hi, I think it depends on what you mean by largest? Most brokers? Producers? Consumers? Messages? Bytes? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Jul 7, 2014 at 6:33 AM, Ersin Er wrote: > Hi, > > LinkedIn

Re: How to parse some of JMX Bean's names.

a point release. > > We would be happy to take a patch from you. How about we look at the size > of the proposed changes and discuss a release timeline on the JIRA? > > Thanks, > Neha > > > On Wed, Jun 4, 2014 at 3:09 PM, Otis Gospodnetic < > otis.gospodne...@gmail.c

Re: How to parse some of JMX Bean's names.

na-Dell-1401353748289-fcaaea29-0-FetchQueueSize" > > > > > > > > Look at part: "spm_new_cluster_topic-af_servers_wawanawna-Dell", what > > is > > > > host name here "servers_wawanawna-Dell" or "wawanawna-Dell&

Re: How to parse some of JMX Bean's names.

Hi Guozhang, On Mon, Jun 2, 2014 at 7:18 PM, Guozhang Wang wrote: > That is indeed a problem, for now, we recommend group name and topic names > to use "_" when there is a need for "-", but this should be fixed > systematically. > Right! For you use case, could you change your topic/group name

Re: kafka-reassign-partitions.sh leaves data on source broker?

licas":[1]}]} Does the above mean, in plain English, "move partitions 0-11 for topic spm_new_cluster_free_system_topic to broker 1"? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, May 31, 20

Re: kafka-reassign-partitions.sh leaves data on source broker?

com/ On Sat, May 31, 2014 at 7:05 PM, Guozhang Wang wrote: > Hello Otis, > > What is your replication factor? > > Guozhang > > > On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi, > > > > I ha

Re: Topic Reassignment Tool Improvements

+1 for sticking this on the Wiki or in JIRA without your internal stuff and letting those who need it (like us) clean up for inclusion. We've got a need for this, too. :) Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

Num of partitions != num.partitions in config

Hi, I have Kafka 0.8.1.1 running and configured to have 24 partitions: $ grep partition /usr/lib/kafka_2.8.0-0.8.1.1/config/server.properties # The default number of log partitions per topic. More partitions allow greater num.partitions=24 But some of the topics don't seem to have 24 partitions,

kafka-reassign-partitions.sh leaves data on source broker?

Hi, I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying to balance them (read: free disk space on the overloaded broker). Is there a tool that will just rebalance everything? I couldn't find one, so I ran this: $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper zk1.prod

Re: stream logs from remote servers.

Hi, Try: https://github.com/joekiller/logstash-kafka https://github.com/rngadam/sendkafka https://issues.apache.org/jira/browse/FLUME-2242 https://github.com/baniuyao/flume-kafka Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

Re: JAVA HEAP settings for KAFKA in production

Hi, We have 2 small EC2 instances acting as "Receivers" for SPM. Each of them runs Apache, Jetty, and a Kafka broker with the following params: java -Xmx512M -server -Dlog4j.configuration=file:/usr/lib/kafka/bin/../config/log4j.properties -cp :/usr/lib/kafka/bin/../proj

Re: Kafka/Zookeeper co-location

Hi, We have Kafka + ZK co-located on EC2 instances. Check this thread from a few weeks ago: http://search-hadoop.com/m/4TaT4Sb6Jg2 Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Apr 24, 2014 at 11:53 AM, Andrew

Re: Kafka integration...

In our case it's straight Java. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Apr 11, 2014 at 8:58 PM, Michael Campbell < michael.campb...@gmail.com> wrote: > Are most of you using straight Java or Scala APIs to tal

Re: Java API to monitor Consumer Offset and Lag

s Solr & Elasticsearch Support * http://sematext.com/ On Sat, Mar 29, 2014 at 1:00 AM, Steve Morin wrote: > Otis, > What's SPM that your referencing? > -Steve > > > On Fri, Mar 28, 2014 at 4:45 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > &

  1   2   >