Re: [kafka-clients] Kafka 0.9.0 release branch

2015-10-12 Thread Ewen Cheslack-Postava
Not sure if I'd call it a blocker, but if we can get it in I would *really* like to see some solution to https://issues.apache.org/jira/browse/KAFKA-2397 committed. Without an explicit leave group, even normal operation of groups can leave some partitions unprocessed for 30-60s at a time under *nor

Kafka 0.9.0 release branch

2015-10-12 Thread Jun Rao
Hi, Everyone, As we are getting closer to the 0.9.0 release, we plan to cut an 0.9.0 release branch in about two weeks from now. In the meantime, we will try to resolve most if not all 0.9.0 blockers listed below. https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20i

Re: Does Kafka recover all data if node is reimaged

2015-10-12 Thread Todd Palino
Yes. As long as you have not reassigned the partitions to other brokers, the wiped broker will rebuild from replication. Keep in mind, however, that if you are using the retention by time configuration, you will have 2x retention on that broker for the length of retention. This means that if your r

Consume messages from offset A to B with the kafka shell scripts, etc.? And get total messages given starting and ending offset?

2015-10-12 Thread David Luu
Hi, Similar to consuming max messages N (or count of messages N in kafkacat), is there any way/option to consume starting from offset A up to ending offset B (we know only offsets but not the total # of messages between those offsets) It looks like that's not available natively? And I guess one w

Partitions - Brokers - Servers

2015-10-12 Thread Julio Camara
I think I understand the concept of partitioning a topic and it's benefits. But I can't find any clear explanation of relationship between partitions, brokers and physical servers. Is there any document or article that discusses the number of partitions for each broker and brokers for each serv

Does Kafka recover all data if node is reimaged

2015-10-12 Thread Rajasekar Elango
I was wondering if a kafka broker node get reimaged and all data is wiped off, Will kafka recover all data on node from replication? -- Thanks, Raja.

Re: New Consumer - discover consumer groups

2015-10-12 Thread Ben Stopford
I double checked with Jun but there is currently no direct API for consumer group discovery. I expect you already know this but you can get a consumer’s offset in the new API. You could also derive the info you need from the offsets topic. B > On 12 Oct 2015, at 17:09, Damian Guy wrote: >

New Consumer - discover consumer groups

2015-10-12 Thread Damian Guy
Hi, Assuming i am using the latest kafka (trunk), exclusively with the new consumer, and i want to monitor consumer lag across all groups - how would i go about discovering the consumer groups? Is there an API call for this? Thanks, Damian

Re: kafka metrics emitting to graphite

2015-10-12 Thread Ian Kallen
I plan on trying this out https://github.com/airbnb/kafka-statsd-metrics2 but yea, JMXTrans is a good option for getting mbean values into graphite as well. On 10/11/15, 8:18 PM, "sunil kalva" wrote: >tx ben, >i though we have native support for graphite in kafka, i will start >exploring thi

Kafka Async producer throwing Exception before configured timeout.

2015-10-12 Thread Madhukar Bharti
Hi , We are getting below exception frequently even, we have configured timeout as 1ms. Does anyone also has faced this issue. org.apache.kafka.common.errors.TimeoutException: Failed to update metadata > after 28 ms We are using Kafka-0.8.2.1 and producer configurations are: > message.se

Re: Getting started with Kafka using Java client

2015-10-12 Thread Ben Stopford
Hi Tarun There is an examples section in the kafka project here which shows the Consumer, SingleConsumer and Producer. These are just clients so you’ll need ZK and a Kafka server running to get them going. You probably don’t need to worry ab

Getting started with Kafka using Java client

2015-10-12 Thread Tarun Sapra
Hi All, Can someone please point me to a well documented github code example for using Kafka 's java client. So far I haven't been able to find much related to using Java client. Most of the examples are using command prompt for producing and consuming messages. It's a bit frustrating that though