Discrepancy between JMX OfflinePartitionCount and kafka-topics.sh?

Hello - I am seeing that the node in our Kafka cluster currently elected as controller is reporting 1 offline partition via JMX (kafka.controller:type=KafkaController,name=OfflinePartitionsCount). However, when I use kaka-topics to find the offline partition (bin/kafka-topics.sh --zookeeper

Re: Discrepancy between JMX OfflinePartitionCount and kafka-topics.sh?

I ran into kind of a similar discrepancy, but about UnderReplicatedPartitions. kafka-topics.sh and zookeeper were saying that we had underreplicated partitions. But JMX said that there were none. I took one of the partitions that ZK was saying was under-replicated and I ran DumpLogSegments on

Re: Kafka Consumer for 0.8.x.x

Could anyone point me to an older version of the consumer client that I could use to run against the 0.8.2 version of Kafka? On Tue, Feb 9, 2016 at 6:57 PM, Ewen Cheslack-Postava wrote: > The new consumer wasn't implemented until 0.9.0.0. The API was sketched > out, but no

Kafka 0.8.2.0 Conusmer example

Could anyone point me to some sample code where I could get some insights on how to write a consumer to do the following: 1. poll at regular intervals 2. get a set of messages during each poll operation 3. process the messages, comit the offset Thanks, Joe

Question about offset expiration

I have a question about the config value offsets.retention.minutes in kafka 0.9.0.0. Is this the timeout for when offsets get compacted in the topic or actually deleted (as it appears from a cursory reading of

Kafka Consumer for 0.8.x.x

Is this intentioal in the Kafka 0.8.2.0 version, org.apache.kafka.clients.consumer.KafkaConsumer, the method: public Map> poll(long timeout) { return null; } Which version of 0.8.x.x should I use so that I could do a consumer.poll(2000)?

Kafka Rest Proxy health check URL

I am looking to monitor my kakfa rest proxy servers and was wondering if there is an endpoint that can be checked to see if the server is running. I am currently checking if endpoint /topics returns a 200. Is that a reasonable endpoint to run the test against. I would prefer to have one that is

Fwd: Re: [kafka-clients] java clients problem

-- Forwarded message -- From: "Dana Powers" Date: Feb 9, 2016 7:59 PM Subject: Re: [kafka-clients] java clients problem To: "iman teimoornegad" Cc: You should email users@kafka.apache.org . this mailing list is used for

Re: Kafka Consumer for 0.8.x.x

If you are working against 0.8.2 brokers and cannot update, you'll have to use the old consumer API for now. There are two parts to it, but you probably want the high level consumer, documented here: http://kafka.apache.org/documentation.html#highlevelconsumerapi -Ewen On Tue, Feb 9, 2016 at

Re: Kafka Consumer for 0.8.x.x

Please check out consumer configs. http://kafka.apache.org/082/documentation.html#consumerconfigs On Tue, Feb 9, 2016 at 1:16 PM, Joe San wrote: > Can I do automatic offset commit using the highlevel consumer? If so, where > is the offset being comitted? > > On Tue, Feb

Re: Kafka Consumer for 0.8.x.x

Auto commit is enabled by default (auto.commit.enable) and Zookeeper is the default storage for compatibility reasons (offsets.storage), but using Kafka instead is recommended. See http://kafka.apache.org/090/documentation.html#oldconsumerconfigs for details on configuring the old consumer.

Re: Kafka Rest Proxy health check URL

Thanks Ewen. Sent from my iPhone > On Feb 9, 2016, at 12:59 PM, Ewen Cheslack-Postava wrote: > > Yeah, you definitely want one that's cheaper than /topics. The root > resource / is effectively a nop you can check for liveness of the service > (but not for, e.g., more

Re: Kafka Consumer for 0.8.x.x

Can I do automatic offset commit using the highlevel consumer? If so, where is the offset being comitted? On Tue, Feb 9, 2016 at 10:13 PM, Ewen Cheslack-Postava wrote: > If you are working against 0.8.2 brokers and cannot update, you'll have to > use the old consumer API for

per topic metrics from new consumer(0.9)

Hi all, We implemented a MetricsReporter to collect metrics from the new producer and the new consumer. For the producer, we did see a bunch of topic specific metrics under "producer-topic-metrics" group. But the consumer seems not providing any per topic metrics. All I saw are consumer-metrics,

Re: per topic metrics from new consumer(0.9)

One thing to add, the old consumer(0.8) does provide per topic metrics and they are under the group, 'ConsumerTopicMetrics'. On Tue, Feb 9, 2016 at 3:20 PM, Yifan Ying wrote: > Hi all, > > We implemented a MetricsReporter to collect metrics from the new producer > and the

Re: Question about offset expiration

Hi Matvey, I have a question about the config value offsets.retention.minutes in kafka > 0.9.0.0. > Is this the timeout for when offsets get compacted in the topic or actually > deleted (as it appears from a cursory reading of > >

Re: Building the 0.9.0 branch

Hi John, The core jar can be built with either Scala 2.10 or Scala 2.11 and the Scala version is included as part of the artifact. The read me page explains this in more detail. Search for "all scala versions" to find out how to run commands for all Scala versions or "particular version of Scala"

Re: Kafka Consumer for 0.8.x.x

How could I use Kafka for offset management with the 0.8.2 version of Kafka` On Tue, Feb 9, 2016 at 10:54 PM, Yifan Ying wrote: > Please check out consumer configs. > http://kafka.apache.org/082/documentation.html#consumerconfigs > > On Tue, Feb 9, 2016 at 1:16 PM, Joe San

Building the 0.9.0 branch

Folks I am trying to work with the latest from the 0.9.0 branch (0.9.0.1-SNAPSHOT) The build appears to have changed the Maven artifact which it produces, at least for the "core" part of the project When I built about a week ago, the Artifact was org.apache.kafka:kafka_2.11:0.9.0.1-SNAPSHOT

Re: Kafka Consumer for 0.8.x.x

The new consumer wasn't implemented until 0.9.0.0. The API was sketched out, but no implementation had been included yet. -Ewen On Tue, Feb 9, 2016 at 8:00 AM, Joe San wrote: > Is this intentioal in the Kafka 0.8.2.0 version, >

Re: Kafka Rest Proxy health check URL

Yeah, you definitely want one that's cheaper than /topics. The root resource / is effectively a nop you can check for liveness of the service (but not for, e.g., more general health like connectivity to ZK or Kafka). The most it will ever do is return a list of available subresources, so it should