Hi Michal, Restart consumer should be easy to implement in a script. The reason not implementing this function inside Kafka consumer is to avoid missing any potential issue/bugs causing consumer to stop.
Guozhang On Mon, Jul 14, 2014 at 1:45 AM, Michal Michalski < [email protected]> wrote: > Hi Guozhang, > > OK, I spent some time to understand a bit more how Kafka uses ZooKeeper and > how sessions are handled and it seems that the change you proposed should > do the job. Thanks :-) > > But I still think that (optional?) automatic restart of a consumer could be > a good idea! ;-) > > M. > > > > Kind regards, > Michał Michalski, > [email protected] > > > On 11 July 2014 16:18, Guozhang Wang <[email protected]> wrote: > > > Hi Michal, > > > > In your case you could try to increase the zookeeper session timeout > value > > on the consumer side (default is 6 sec) and see if this is sufficient to > > cover the latency jitters. > > > > Guozhang > > > > > > On Fri, Jul 11, 2014 at 5:25 AM, Michal Michalski < > > [email protected]> wrote: > > > > > Hey Guozhang, > > > > > > Thanks for reply. I get your point on "hiding" some issues, but I'd > > prefer > > > to separate the recovery and reporting a failure. Also, I think if > simple > > > restart is a possible solution, it shouldn't require implementing it > > > separately or, what's even worse, a manual intervention. Maybe I'll > > > describe my problem then to show you my point of view: > > > > > > ZK latency spiked for few seconds making ZK effectively dead from > > > consumers' point of view. Then they all reconnected. As I understand, > > when > > > it happened, it caused rebalancing. Some consumer groups succeeded, but > > > then another spike in latency happened and - as we suspect - it caused > > > rebalancing to fail, because creation of that ZK node failed at some > > point. > > > Ideally, I'd like to get notified about that problem (rebalancing > failed > > > after X retries etc.), so I know there is an issue and I can > investigate > > > it, but then I'd like Kafka consumer (or my app) to fallback to > restart, > > > which could *possibly* make consumer recover. If not - that's my > problem > > > then ;-) > > > > > > In our case it was enough to restart the app to get consumer working > > again, > > > but - as we didn't know about that behaviour before and we weren't > > prepared > > > for it - it required manual intervention (on Friday night, which made > it > > > even more painful ;> ) which, we believe, wasn't necessary in that case > > and > > > could have been handled automatically. > > > > > > M. > > > > > > > > > > > > Kind regards, > > > Michał Michalski, > > > [email protected] > > > > > > > > > On 10 July 2014 23:43, Guozhang Wang <[email protected]> wrote: > > > > > > > Hi Michal, > > > > > > > > The rebalance will only be triggered on consumer membership or > > > > topic/partition changes. Once triggered it will try to finish the > > > rebalance > > > > for at most rebalance.max.retries times, i.e. if it fails it will > wait > > > for > > > > rebalance.backoff.ms, and then try again until number of retries > > > > exhausted. > > > > When it happens an exception will be thrown and the consumer may be > > > fallen > > > > to a bad state. > > > > > > > > Then reason we did not implement automatic restart upon rebalance > > > failures > > > > is that it may actually "hide" some issues in the systems that > actually > > > > caused the rebalance failure. The general design is that if some > > > > exception/errors are not expected like the rebalance failures we will > > let > > > > it to possibly hault/kill the instance rather than automatically > > restart > > > > and let it go. > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > On Thu, Jul 10, 2014 at 2:24 AM, Michal Michalski < > > > > [email protected]> wrote: > > > > > > > > > Hi, > > > > > > > > > > Just wondering - is there any reason why rebalance.max.retries is 4 > > by > > > > > default? Is there any good reason why I shouldn't expect my > consumers > > > to > > > > > keep trying to rebalance for minutes (e.g. 30 retries every 6 > > seconds), > > > > > rather than seconds (4 retries every 2 seconds by default)? > > > > > > > > > > Also, if my consumer fails to rebalance because of NoNodeException > > > > > (org.apache.zookeeper.KeeperException$NoNodeException: > > KeeperErrorCode > > > = > > > > > NoNode for > > > /consumers/is-entity-modified-document-group/ids/<something>) > > > > > wouldn't that make sense to make Kafka restart it automatically > once > > it > > > > > "uses" all the retries attempts? Or recreate the inexistent ZK node > > > > like, I > > > > > believe, it will happen on consumer restart? > > > > > > > > > > I'm asking because that kind of errors seem to be "recoverable" > ones, > > > > but - > > > > > if I understand it correctly - with current design they require > > > > > implementing additional mechanisms or manual intervention. > > > > > > > > > > > > > > > Kind regards, > > > > > Michał > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
