Chen, >From ZK logs it sounds like ZK kept timed out consumers which triggers rebalance.
What is the zk session timeout config value in your consumers? Guozhang On Thu, Nov 13, 2014 at 10:15 AM, Chen Wang <[email protected]> wrote: > Thanks for the info. > It makes sense, however, I didn't see any "session timeout"/"expired" > entries in consumer log.. > but do see lots of connection closed entry in zookeeper log: > > 2014-11-13 10:07:53,132 [myid:1] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for > client /10.93.83.50:37180 which had sessionid 0x149a4cc1b580e7d > 2014-11-13 10:08:04,499 [myid:1] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@197] - Accepted socket > connection > from /10.93.80.121:38437 > 2014-11-13 10:08:04,503 [myid:1] - WARN [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2182:ZooKeeperServer@822] - Connection request from old > client /10.93.80.121:38437; will be dropped if server is in r-o mode > 2014-11-13 10:08:04,503 [myid:1] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2182:ZooKeeperServer@868] - Client attempting to establish > new session at /10.93.80.121:38437 > 2014-11-13 10:08:04,538 [myid:1] - INFO > [CommitProcessor:1:ZooKeeperServer@617] - Established session > 0x149a4cc1b580e7e with negotiated timeout 40000 for client / > 10.93.80.121:38437 > 2014-11-13 10:08:08,746 [myid:1] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for > client /10.93.80.121:38437 which had sessionid 0x149a4cc1b580e7e > > We are using -Xmx2048m for consumer, and I didn't see any GC related > exceptions > > Chen > > > > On Thu, Nov 13, 2014 at 9:13 AM, Guozhang Wang <[email protected]> wrote: > > > Hey Chen, > > > > As Neha suggested, typical reason of too many rebalances is that your > > consumers kept being timed out from ZK, and you can verify this by > checking > > in your consumer logs for sth. like "session timeout" entries (these are > > not ERROR entries). > > > > Guozhang > > > > Guozhang > > > > On Wed, Nov 12, 2014 at 5:31 PM, Neha Narkhede <[email protected]> > > wrote: > > > > > Does this help? > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog > > > ? > > > > > > On Wed, Nov 12, 2014 at 3:53 PM, Chen Wang <[email protected] > > > > > wrote: > > > > > > > Hi there, > > > > My kafka client is reading a 3 partition topic from kafka with 3 > > threads > > > > distributed on different machines. I am seeing frequent owner changes > > on > > > > the topics when running: > > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > > > > my_test_group --topic mytopic -zkconnect localhost:2181 > > > > > > > > The owner kept changing once a while, but I didn't see any exceptions > > > > thrown from the consumer side. When checking broker log, its full of > > > > INFO Closing socket connection to /IP. (kafka.network.Processor) > > > > > > > > Is this expected behavior? If so, how can I tell when the leader is > > > > imbalanced, and rebalance is triggered? > > > > Thanks, > > > > Chen > > > > > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
