[jira] Commented: (ZOOKEEPER-759) Stop accepting connections when close to file descriptor limit
[ https://issues.apache.org/jira/browse/ZOOKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873178#action_12873178 ] Savu Andrei commented on ZOOKEEPER-759: --- I have updated ZOOKEEPER-744 to export the current number of open file descriptors and the maximum number allowed. I'm still not sure how I should implement zk_max_file_descriptor_mismatch. I'll look more into this in the near future. > Stop accepting connections when close to file descriptor limit > -- > > Key: ZOOKEEPER-759 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759 > Project: Zookeeper > Issue Type: Improvement > Components: server >Reporter: Travis Crawford > Fix For: 3.4.0 > > > Zookeeper always tries to accept new connections, throwing an exception if > out of file descriptors. An improvement would be denying new client > connections when close to the limit. > Additionally, file-descriptor limits+usage should be exported to the > monitoring four-letter word, should that get implemented (see ZOOKEEPER-744). > DETAILS > A Zookeeper ensemble I administer recently suffered an outage when one node > was restarted with the low system-default ulimit of 1024 file descriptors and > later ran out. File descriptor usage+max are already being monitored by the > following MBeans: > - java.lang.OperatingSystem.MaxFileDescriptorCount > - java.lang.OperatingSystem.OpenFileDescriptorCount > They're described (rather tersely) at: > http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html > This feature request is for the following: > (a) Stop accepting new connections when OpenFileDescriptorCount is close to > MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be > denied, logged to disk at debug level, and increment a > ``ConnectionDeniedCount`` MBean counter. > (b) Begin accepting new connections when usage drops below some configurable > threshold, defaulting to 90% of FD usage, basically the high/low watermark > model. > (c) Update the administrators guide with a comment about using an appropriate > FD limit. > (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for: > zookeeper_open_file_descriptor_count > zookeeper_max_file_descriptor_count > zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not > all zk's have the same max FD value -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-759) Stop accepting connections when close to file descriptor limit
[ https://issues.apache.org/jira/browse/ZOOKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862320#action_12862320 ] Patrick Hunt commented on ZOOKEEPER-759: Is there anything comparable on the windows jvm, or no options at all on that platform? What about alternative (non sun) jvms? Guess we treat those similar to windows... we should think about it when implementing though. > Stop accepting connections when close to file descriptor limit > -- > > Key: ZOOKEEPER-759 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759 > Project: Zookeeper > Issue Type: Improvement > Components: server >Reporter: Travis Crawford > > Zookeeper always tries to accept new connections, throwing an exception if > out of file descriptors. An improvement would be denying new client > connections when close to the limit. > Additionally, file-descriptor limits+usage should be exported to the > monitoring four-letter word, should that get implemented (see ZOOKEEPER-744). > DETAILS > A Zookeeper ensemble I administer recently suffered an outage when one node > was restarted with the low system-default ulimit of 1024 file descriptors and > later ran out. File descriptor usage+max are already being monitored by the > following MBeans: > - java.lang.OperatingSystem.MaxFileDescriptorCount > - java.lang.OperatingSystem.OpenFileDescriptorCount > They're described (rather tersely) at: > http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html > This feature request is for the following: > (a) Stop accepting new connections when OpenFileDescriptorCount is close to > MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be > denied, logged to disk at debug level, and increment a > ``ConnectionDeniedCount`` MBean counter. > (b) Begin accepting new connections when usage drops below some configurable > threshold, defaulting to 90% of FD usage, basically the high/low watermark > model. > (c) Update the administrators guide with a comment about using an appropriate > FD limit. > (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for: > zookeeper_open_file_descriptor_count > zookeeper_max_file_descriptor_count > zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not > all zk's have the same max FD value -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-759) Stop accepting connections when close to file descriptor limit
[ https://issues.apache.org/jira/browse/ZOOKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862307#action_12862307 ] Mahadev konar commented on ZOOKEEPER-759: - was just goint to comment on similar lines ted! yeah we should make sure this case is taken care of... > Stop accepting connections when close to file descriptor limit > -- > > Key: ZOOKEEPER-759 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759 > Project: Zookeeper > Issue Type: Improvement > Components: server >Reporter: Travis Crawford > > Zookeeper always tries to accept new connections, throwing an exception if > out of file descriptors. An improvement would be denying new client > connections when close to the limit. > Additionally, file-descriptor limits+usage should be exported to the > monitoring four-letter word, should that get implemented (see ZOOKEEPER-744). > DETAILS > A Zookeeper ensemble I administer recently suffered an outage when one node > was restarted with the low system-default ulimit of 1024 file descriptors and > later ran out. File descriptor usage+max are already being monitored by the > following MBeans: > - java.lang.OperatingSystem.MaxFileDescriptorCount > - java.lang.OperatingSystem.OpenFileDescriptorCount > They're described (rather tersely) at: > http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html > This feature request is for the following: > (a) Stop accepting new connections when OpenFileDescriptorCount is close to > MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be > denied, logged to disk at debug level, and increment a > ``ConnectionDeniedCount`` MBean counter. > (b) Begin accepting new connections when usage drops below some configurable > threshold, defaulting to 90% of FD usage, basically the high/low watermark > model. > (c) Update the administrators guide with a comment about using an appropriate > FD limit. > (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for: > zookeeper_open_file_descriptor_count > zookeeper_max_file_descriptor_count > zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not > all zk's have the same max FD value -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-759) Stop accepting connections when close to file descriptor limit
[ https://issues.apache.org/jira/browse/ZOOKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862303#action_12862303 ] Ted Dunning commented on ZOOKEEPER-759: --- This is a unix specific bean so don't forget to defang the test if the bean isn't available. > Stop accepting connections when close to file descriptor limit > -- > > Key: ZOOKEEPER-759 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759 > Project: Zookeeper > Issue Type: Improvement > Components: server >Reporter: Travis Crawford > > Zookeeper always tries to accept new connections, throwing an exception if > out of file descriptors. An improvement would be denying new client > connections when close to the limit. > Additionally, file-descriptor limits+usage should be exported to the > monitoring four-letter word, should that get implemented (see ZOOKEEPER-744). > DETAILS > A Zookeeper ensemble I administer recently suffered an outage when one node > was restarted with the low system-default ulimit of 1024 file descriptors and > later ran out. File descriptor usage+max are already being monitored by the > following MBeans: > - java.lang.OperatingSystem.MaxFileDescriptorCount > - java.lang.OperatingSystem.OpenFileDescriptorCount > They're described (rather tersely) at: > http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html > This feature request is for the following: > (a) Stop accepting new connections when OpenFileDescriptorCount is close to > MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be > denied, logged to disk at debug level, and increment a > ``ConnectionDeniedCount`` MBean counter. > (b) Begin accepting new connections when usage drops below some configurable > threshold, defaulting to 90% of FD usage, basically the high/low watermark > model. > (c) Update the administrators guide with a comment about using an appropriate > FD limit. > (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for: > zookeeper_open_file_descriptor_count > zookeeper_max_file_descriptor_count > zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not > all zk's have the same max FD value -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-759) Stop accepting connections when close to file descriptor limit
[ https://issues.apache.org/jira/browse/ZOOKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862301#action_12862301 ] Patrick Hunt commented on ZOOKEEPER-759: Allowing the user to override entirely seems like a good feature (config) as well. I always like to support this with new features that have potentially significant downside, just in case we overlooked something during concept/design/dev. > Stop accepting connections when close to file descriptor limit > -- > > Key: ZOOKEEPER-759 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759 > Project: Zookeeper > Issue Type: Improvement > Components: server >Reporter: Travis Crawford > > Zookeeper always tries to accept new connections, throwing an exception if > out of file descriptors. An improvement would be denying new client > connections when close to the limit. > Additionally, file-descriptor limits+usage should be exported to the > monitoring four-letter word, should that get implemented (see ZOOKEEPER-744). > DETAILS > A Zookeeper ensemble I administer recently suffered an outage when one node > was restarted with the low system-default ulimit of 1024 file descriptors and > later ran out. File descriptor usage+max are already being monitored by the > following MBeans: > - java.lang.OperatingSystem.MaxFileDescriptorCount > - java.lang.OperatingSystem.OpenFileDescriptorCount > They're described (rather tersely) at: > http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html > This feature request is for the following: > (a) Stop accepting new connections when OpenFileDescriptorCount is close to > MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be > denied, logged to disk at debug level, and increment a > ``ConnectionDeniedCount`` MBean counter. > (b) Begin accepting new connections when usage drops below some configurable > threshold, defaulting to 90% of FD usage, basically the high/low watermark > model. > (c) Update the administrators guide with a comment about using an appropriate > FD limit. > (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for: > zookeeper_open_file_descriptor_count > zookeeper_max_file_descriptor_count > zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not > all zk's have the same max FD value -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-759) Stop accepting connections when close to file descriptor limit
[ https://issues.apache.org/jira/browse/ZOOKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862296#action_12862296 ] Patrick Hunt commented on ZOOKEEPER-759: You allude to it but to be specific we should make the thresholds configurable (zoo.cfg) with sensible defaults. > Stop accepting connections when close to file descriptor limit > -- > > Key: ZOOKEEPER-759 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759 > Project: Zookeeper > Issue Type: Improvement > Components: server >Reporter: Travis Crawford > > Zookeeper always tries to accept new connections, throwing an exception if > out of file descriptors. An improvement would be denying new client > connections when close to the limit. > Additionally, file-descriptor limits+usage should be exported to the > monitoring four-letter word, should that get implemented (see ZOOKEEPER-744). > DETAILS > A Zookeeper ensemble I administer recently suffered an outage when one node > was restarted with the low system-default ulimit of 1024 file descriptors and > later ran out. File descriptor usage+max are already being monitored by the > following MBeans: > - java.lang.OperatingSystem.MaxFileDescriptorCount > - java.lang.OperatingSystem.OpenFileDescriptorCount > They're described (rather tersely) at: > http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html > This feature request is for the following: > (a) Stop accepting new connections when OpenFileDescriptorCount is close to > MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be > denied, logged to disk at debug level, and increment a > ``ConnectionDeniedCount`` MBean counter. > (b) Begin accepting new connections when usage drops below some configurable > threshold, defaulting to 90% of FD usage, basically the high/low watermark > model. > (c) Update the administrators guide with a comment about using an appropriate > FD limit. > (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for: > zookeeper_open_file_descriptor_count > zookeeper_max_file_descriptor_count > zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not > all zk's have the same max FD value -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
