Fwd: Pulling metrics from kafka-console-producer.

-- Forwarded message -- From: Pavan Kumar Date: Tue, Sep 8, 2015 at 3:23 PM Subject: Pulling metrics from kafka-console-producer. To: jmxtrans Hi, I have been trying to pull metrics from kafka-console-producer with jmxtrans as follows : * --> bin/kafka-run-class kafka

Issue in pulling metrics from kafka-console-producer.

*I have installed JMXReporter to pull metrics from Apache Kafka-2.11-0.8.2.0 which is extracting allthe Server Metrics as seen in page http://docs.confluent.io/1.0/kafka/monitoring.html and not the Producer and Consumer Metrics. Is there any way

Fwd: Issue in pulling metrics from kafka-console-producer.

-- Forwarded message -- From: Pavan Kumar Date: Thu, Sep 10, 2015 at 12:16 PM Subject: Issue in pulling metrics from kafka-console-producer. To: users@kafka.apache.org *I have installed JMXReporter to pull metrics from Apache Kafka-2.11-0.8.2.0 which is extracting allthe Serv

Re: Resetting consumer offsets after moving to offset.storage=kafka

Thank you very much, Erik. Yes, the SimpleConsumer definitely would achieve the goal. It was easier with ImportZkOffset, as it only takes a couple of lines of commands. If no equivalent of ImportZkOffset, I will go with the SimpleConsumer. As for the data loss, it took place in the downstream pr

Re: Resetting consumer offsets after moving to offset.storage=kafka

It is possible to commit offsets using the SimpleConsumer API to kafka or zookeeper for any GroupID, topic, and partition tuple. There are some difficulties with the SimpleConsumer, but it should be able to make the call within your app. See the scala Doc here: http://apache.mirrorcatalogs.com/ka

Resetting consumer offsets after moving to offset.storage=kafka

Hi, We have a consumer that under certain circumstances may lose data. To guard against such data loss, we have a tool that periodically pulls and stores offsets from zk. Once a data loss takes place, we use our historical offsets to reset the consumer offset on zk. With offset.storage=zookeepe

Re: latency test

Thank you Erik. In my test I am using fixed 200bytes messages and I run 500k messages per producer on 92 physically isolated producers. Each test run takes about 20 minutes. As the broker cluster is migrated into a new physical cluster, I will perform my test and get the latency results in the nex

Re: latency test

Yes, and that can really hurt average performance. All the partitions were nearly identical up to the 99%’ile, and had very good performance at that level hovering around a few milli’s. But when looking beyond the 99%’ile, there was that clear fork in the distribution where a set of 3 partitions

Re: latency test

So are you suggesting that the long delays happened in %1 percentile happens in the slower partitions that are further away? Thanks. On Wed, Sep 9, 2015 at 3:15 PM, Helleren, Erik wrote: > So, I did my own latency test on a cluster of 3 nodes, and there is a > significant difference around the 9

Re: latency test

So, I did my own latency test on a cluster of 3 nodes, and there is a significant difference around the 99%’ile and higher for partitions when measuring the the ack time when configured for a single ack. The graph that I wish I could attach or post clearly shows that around 1/3 of the partitions s

Re: [VOTE] 0.8.2.2 Candidate 1

Thanks everyone for voting. The following are the results of the votes. +1 binding = 3 votes +1 non-binding = 3 votes -1 = 0 votes 0 = 0 votes The vote passes. I will release artifacts to maven central, update the dist svn and download site. Will send out an announce after that. Thanks everyon

Re: [VOTE] 0.8.2.2 Candidate 1

+1 from me too. Jun On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.8.2.2. This only > fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in > 0.8.2.1. > > Release Notes for the 0.8.2.2 release > > https://people.ap

Re: [kafka-clients] [VOTE] 0.8.2.2 Candidate 1

+1 binding On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.8.2.2. This only > fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in > 0.8.2.1. > > Release Notes for the 0.8.2.2 release > https://people.apache.org/~jun

Re: [VOTE] 0.8.2.2 Candidate 1

+1 non-binding - verified signatures and build. On Wed, Sep 9, 2015 at 10:28 AM, Ewen Cheslack-Postava wrote: > +1 non-binding. Verified artifacts, unit tests, quick start. > > On Wed, Sep 9, 2015 at 10:09 AM, Guozhang Wang wrote: > > > +1 binding, verified unit tests and quick start. > > > > O

Re: [VOTE] 0.8.2.2 Candidate 1

+1 non-binding. Verified artifacts, unit tests, quick start. On Wed, Sep 9, 2015 at 10:09 AM, Guozhang Wang wrote: > +1 binding, verified unit tests and quick start. > > On Wed, Sep 9, 2015 at 4:12 AM, Manikumar Reddy > wrote: > > > +1 (non-binding). verified the artifacts, quick start. > > > >

Re: some producers stuck when one broker is bad

1) any suggestion on how to identify the bad broker(s)? ---> At Linkedin we have alerts that are setup using our internal scripts for detecting if a broker has gone bad. We also check the under replicated partitions and that can tell us which broker has gone bad. By broker going bad, it can mean di

Re: async producer callback not reliable

Make sure you have inflight requests set to 1 if you want ordered messages. Thanks, Mayuresh On Tue, Sep 8, 2015 at 5:55 AM, Damian Guy wrote: > Can you do: > producer.send(...) > ... > producer.send(...) > producer.flush() > > By the time the flush returns all of your messages should have bee

Re: [VOTE] 0.8.2.2 Candidate 1

+1 binding, verified unit tests and quick start. On Wed, Sep 9, 2015 at 4:12 AM, Manikumar Reddy wrote: > +1 (non-binding). verified the artifacts, quick start. > > On Wed, Sep 9, 2015 at 2:41 AM, Ashish wrote: > > > +1 (non-binding) > > > > Ran the build, works fine. All test cases passed > >

MirrorMaker - Not consuming from all partitions

Hello, Hope everyone is doing well. I was hoping to get some assistance with a strange issue we're experiencing while using the MirrorMaker to pull data down from an 8 node Kafka cluster in AWS into our data center. Both Kafka clusters and the mirror are using version 0.8.1.1 with dedicated Zookee

Re: [VOTE] 0.8.2.2 Candidate 1

+1 (non-binding). verified the artifacts, quick start. On Wed, Sep 9, 2015 at 2:41 AM, Ashish wrote: > +1 (non-binding) > > Ran the build, works fine. All test cases passed > > On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao wrote: > > This is the first candidate for release of Apache Kafka 0.8.2.2. Th