Issues due to Running multiple Kafka clusters using the same zookeeper service

2017-08-11 Thread Raghav
Hi We are chrooting each Kafka cluster to a unique namespace in the Kafka brokers of each our clusters. We can produce the message fine and we can write to broker. But our consumers are not able to consume anything from their respective brokers. We are seeing the following error messages after ena

Re: Kafka Producer Errors

2017-08-11 Thread Saladi Naidu
Looks like some of the fonts/characters did not come through, am re-sending with correct details We have recently upgraded our Kafka cluster from 0.8.2 to 0.10.1. As part of the upgrade our clients upgraded the client libraries as well. After the upgrade we are facing following issue org.apache.

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

2017-08-11 Thread Viktor Somogyi
Hi Chao, Apart from the actual request, the consumer sends metadata requests as well. What is actually going through the network when you get these latencies? Try increasing the metadata max age with metadata.max.age.ms. (Disclaimer: I myself never ran any benchmarking, just trying to help :). )

Re: Kafka (0.9) ConsumerRebalanceListener onPartitionsRevoked not called.

2017-08-11 Thread Naik, Ninad
Anyone? From: Ninad Naik mailto:ninad.n...@epsilon.com>> Date: Friday, August 4, 2017 at 1:53 PM To: "users@kafka.apache.org" mailto:users@kafka.apache.org>> Subject: Kafka (0.9) ConsumerRebalanceListener onPartitionsRevoked not called. Hello, We’re using custom st

Re: Consumer stops consuming after one of the broker stopped

2017-08-11 Thread UMESH CHAUDHARY
Are you seeing any errors in consumer when your broker gets down? May be any debug messages would be helpful. After killing the broker do you notice any leadership change for those topic's partitions using [kafka-topics.sh --zookeepr <> --describe] ? On Fri, 11 Aug 2017 at 06:23 Rakesh Singh wro

Re: [DISCUSS] Streams DSL/StateStore Refactoring

2017-08-11 Thread Jan Filipiak
Inline rather sparse for the lack of time. Sadly I can't agree to any of your arguments and I _hate_ how its gonna look, but we can't have this discussion for ever. I think I explained everything in enough detail so my points can make sense. if someone is interested and has specific questions

[kafka streams] discuss: dynamically update subscription pattern

2017-08-11 Thread Bart Vercammen
Hi, I have a question basically on how it would be the best way to implement something within Kafka Streams. The thing I would like to do: "dynamically update the subscription pattern of the source topics. The reasoning behind this (in my project): meta data about the source topics is evented on