Stop accepting connections when close to file descriptor limit
--------------------------------------------------------------
Key: ZOOKEEPER-759
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759
Project: Zookeeper
Issue Type: Improvement
Components: server
Reporter: Travis Crawford
Zookeeper always tries to accept new connections, throwing an exception if out
of file descriptors. An improvement would be denying new client connections
when close to the limit.
Additionally, file-descriptor limits+usage should be exported to the monitoring
four-letter word, should that get implemented (see ZOOKEEPER-744).
DETAILS
A Zookeeper ensemble I administer recently suffered an outage when one node was
restarted with the low system-default ulimit of 1024 file descriptors and later
ran out. File descriptor usage+max are already being monitored by the following
MBeans:
- java.lang.OperatingSystem.MaxFileDescriptorCount
- java.lang.OperatingSystem.OpenFileDescriptorCount
They're described (rather tersely) at:
http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html
This feature request is for the following:
(a) Stop accepting new connections when OpenFileDescriptorCount is close to
MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be
denied, logged to disk at debug level, and increment a
``ConnectionDeniedCount`` MBean counter.
(b) Begin accepting new connections when usage drops below some configurable
threshold, defaulting to 90% of FD usage, basically the high/low watermark
model.
(c) Update the administrators guide with a comment about using an appropriate
FD limit.
(d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for:
zookeeper_open_file_descriptor_count
zookeeper_max_file_descriptor_count
zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not
all zk's have the same max FD value
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
