[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford updated ZOOKEEPER-856: -------------------------------------- Attachment: zk_open_file_descriptor_count_total.gif zk_open_file_descriptor_count_members.gif Attached are two graphs showing: - Total ZooKeeper connections to a 3 node cluster - Connections per member in the cluster In the totals graph, notice how its largely unchanged over time. This period represents a steady-state period of usage. In the members graph, notice how the number of connections is significantly different between machines. This cluster allows the leader to service reads, so that's not something to factor in when interpreting number of connections. These graphs look very similar to an issue I had with another service (scribe) and we solved the issue by disconnecting every N+-K messages. We tried getting fancy by publishing load metrics and using a smart selection algorithm. Turns out in practice though the periodic disconnect/reconnect was easier to implement and worked better, so I'm tossing that idea out as a potential solution here. > Connection imbalance leads to overloaded ZK instances > ----------------------------------------------------- > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug > Reporter: Travis Crawford > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.