Kafka client troubles when restarting a broker

Hi guys, When broker goes down due to restart, running and new consumers starting to fail with: [Consumer clientId=consumer-42169, groupId=] Connection to node 67108872 could not be established. Broker may not be available. [Consumer clientId=consumer-46213, groupId=] Connection to node -4 could n

Re: Kafka streams consumer/producer throttling

gt; > Hello Andrey, > > Could you provide a bit more information on how you set the bandwidth based > on client id? Sharing some code snippet would even better for me to > understand your encountered issue. > > Guozhang > > On Fri, Nov 9, 2018 at 7:11 AM Andrey Dyachkov >

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

You can attach EBS volume, which will store data and metadata(e.g. broker id), and then attach it to the new AWS instance and start Kafka, it will pick the broker id plus you won’t need to rebalance the cluster. On Wed 14. Nov 2018 at 19:48, naresh Goud wrote: > Static IP. Buying static IP may h

Kafka streams consumer/producer throttling

Hello, Could you tell me if there is a way to throttle kafka streams by client id? I’ve tried to set bandwidth limit for client id, which is specefied in kafka stream config, but it does not work. -- With great enthusiasm, Andrey

Re: Kafka with Zookeeper behind AWS ELB

e moment I can use an HAproxy with EIP and when the bug is solved I > can move to ELB. > What do you think about it? > Regards > L. > > On Wed, Jul 19, 2017 at 2:16 PM, Andrey Dyachkov < > andrey.dyach...@gmail.com> > wrote: > > > Hi, > > I have just

Re: Kafka with Zookeeper behind AWS ELB

Hi, I have just posted almost the same question in dev list. Zookeeper client resolves address only once, on start, introducing ELB won't really help here (ELBs can be replaced, which involved ip change), but I am eager to know if there is a solution for that. On Wed, 19 Jul 2017 at 23:08 Luigi Ta

Kafka high cpu usage

Hi, Our production env uses Kafka 0.9.0.1 cluster of 12 m3.large nodes. Partitions count per broker is ~450, percent of leaders per broker is 30-40%. The average messages load is ~3K/s, bytes flow in is ~10MB/s and bytes flow out is ~60 MB/s. We observed strange behaviour while putting one instan

Re: Kafka client circuit breaker

n when broker is not available? > > > > On Sun, Oct 30, 2016 at 12:25 PM, Andrey Dyachkov < > andrey.dyach...@gmail.com > >> wrote: > > > >> R Krishna, > >> > >> We have default timeout for producer and consumer it is 30 secs. > >&

Re: Kafka client circuit breaker

or us very important to stay live even if Kafka is unavailable (it happens) that's why I think avoiding cascading issues will be very useful. On Sat, 29 Oct 2016 at 19:22 R Krishna wrote: > What are the issues and problems you want to solve though? > On Oct 29, 2016 3:59 AM, "

Kafka client circuit breaker

Hi, We use Kafka for around half a year as a main backend system for storing events from big variety of internal microservices. In general It works quite well and we experience issues only in some rare 'corner cases' where we either run against a bug or do not understand why Kafka behaves in parti