Well, it's possible that when processing, it might take longer than the zookeeper timeout to process a message, intermittently. Would that cause a zookeeper timeout?
(btw I'm usind 0.8.1.1). On Thu, Aug 7, 2014 at 2:30 AM, Clark Haskins <chask...@linkedin.com.invalid > wrote: > Is your application possibly timing out its zookeeper connection during > consumption while doing its processing, thus triggering the rebalance? > > -Clark > > On 8/6/14, 11:18 PM, "Jason Rosenberg" <j...@squareup.com> wrote: > > >We've noticed that some of our consumers are more likely to repeatedly > >trigger rebalancing when the app is consuming messages more slowly (e.g. > >persisting data to back-end systems, etc.). > > > >If on the other hand we 'fast-forward' the consumer (which essentially > >means we tell it to consume but do nothing with the messages until all > >caught up), it will never decide to do a rebalance during this time. So > >it > >can go hours without rebalancing while fast forwarding and consuming super > >fast, while during normal processing, it might decide to rebalance every > >minute or so. > > > >Is there any simple explanation for this? > > > >Usually the trigger for rebalance logged is that a "topic info for path X > >has changed to Y, triggering rebalance". > > > >Thanks for any ideas. > > > >We'd like to reduce the rebalancing, as it essentially slows down > >consumption each time it happens. > > > >Thanks > > > >Jason > >
