Hi Becket, I asked the same question in the user mailing list and we had a small discussion there.
We have default timeout for producer and consumer it is 30 secs. If Kafka is not available we will wait 30 secs on each request. Decreasing timeout won't help much because we have a lot requests but circuiting them will help us tremendous. We could implement such logic before actually calling Kafka client but there is problem what the trigger is to open the circuit. I thought about circuiting by unavailable broker but it is quite an expansive operation to figure out in the client's code on the flip side Kaka client already does all the job to make out the broker. For us very important to stay live even if Kafka is unavailable (it happens) that's why I think avoiding cascading issues will be very useful. The circuit breaker here is a thing which can stop requests to Kafka for certain period of time until a specific topic/partition/broker is not available. On Mon, 31 Oct 2016 at 21:38 Becket Qin <becket....@gmail.com> wrote: Hi Andrey, I am not sure I understand the question "why doesn't Kafka client have the solution for this kind of problems?" Can you elaborate a little on what kind of problems are you referring to? And what do you mean by "circuit breaker" here? Jiangjie (Becket) Qin On Fri, Oct 28, 2016 at 12:07 PM, Andrey Dyachkov <andrey.dyach...@gmail.com > wrote: > Hi Kafka developers :) > > We use Kafka for around half a year as a main backend system for storing > events from big variety of internal microservices. In general It works > quite well and we experience issues only in some rare 'corner cases' where > we either run against a bug or do not understand why Kafka behaves in > particular manner. The fact Kafka is the main system is used under our > software If it fails our system will fail too. That's why I am writing to > ask about the solution which can help us tolerate different issues on Kafka > side. First of all could you tell me why doesn't Kafka client have the > solution for this kind of problems? Have you ever had thoughts about it? > Thank you in advance! > -- > > With great enthusiasm, > Andrey > -- With great enthusiasm, Andrey
