[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Reed updated ZOOKEEPER-336: Status: Open (was: Patch Available) collision with ZOOKEEPER-431 single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Reed updated ZOOKEEPER-336: Status: Patch Available (was: Open) single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Robinson updated ZOOKEEPER-336: - Attachment: ZOOKEEPER-336.patch I've included a C and Java client test in this new patch. I have modified ZooKeeperServerMain to accept a further optional maxcnxns argument on the command line in order to be able to write the C test. Both client tests work by having two clients connect to a server that only accepts one connection at a time. When the first client connects it is closed, allowing the second to connect as it retries. single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Robinson updated ZOOKEEPER-336: - Status: Patch Available (was: Open) single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-336: --- Status: Open (was: Patch Available) single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-336: --- Status: Patch Available (was: Open) resubmitting the patch - the test failure was cause by : java.net.BindException: Address already in use in one of the tests - as Henry suspected it's a known problem. We have a fix for this, but not yet committed (probably 3.3, it's part of the testng changes). single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Paiva Junqueira updated ZOOKEEPER-336: - Status: Open (was: Patch Available) single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Robinson updated ZOOKEEPER-336: - Attachment: ZOOKEEPER-336.patch Here's another pass at the patch, with a better-written test and updated documentation. single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Robinson updated ZOOKEEPER-336: - Status: Patch Available (was: Open) single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-336: --- Status: Open (was: Patch Available) I noticed the following issues: 1) it's great to see tests ;-) however the included test needs to be improved a bit. Two issues: a) under heavy load (like we see on hudson.zones.apache.org) the test may fail due to the use of sleep. We've found that using sleep in tests is a two fold problem. What happens is that usually the test runs fine, however under heavy load the sleep may not be long enough, resulting in invalid test failure. This is particularly onerous on hudson.zones.apache.org, which is a virtualized system and gates commits. A second reason we generally dont use sleep is that it artificially inflates the run time of the test, making agile/CI based development painful (granted this is something we need to improve on, ie test run time, and are working towards) Instead of using sleep the test should first loop to start all the sockets, then loop to see if the sockets are either connected or failed to connect, counting the number of success and checking against expected. The check should wait for a socket result for some max period of time and if the time is exceeded (say 60sec) then the test fails. b) we should really test the zk client code to ensure they handle this correctly/gracefully. It would be great to have an additional Java and an additional C test that uses ZooKeeper objects and ensures that the first N max work properly when N max clients are started, and that the clients max give proper watcher notification. 2) this is a user externally visible change - the docs need to be updated. Hadoop uses forrest for docs, zookeeper in particular uses simpledocbook xml format for the doc source markup. See the source docs in src/docs/src/documentation/content/xdocs You can re-generate the documentation using forrest, which you can download from http://forrest.apache.org/ Note: forrest requires jdk5 I run forrest as follows: PATH=$PATH:/home/phunt/dev/apache-forrest-0.8/bin JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun ant -Dforrest.home=/home/phunt/dev/apache-forrest-0.8 -Djava5.home=/usr/lib/jvm/java-1.5.0-sun docs the generated docs are put into trunk/docs. You probably want to put this into the admin/ops docs. 3) in getClientCnxnCount there's no need for removeall (inefficient as well), just use an int and count unclosed 4) in the updated patch I changed info to warning for the max connection exceeded. This happens infrequently enough, and it's important enough to list as a warning. 5) Note: we have 80 character limit on line length. the updated patch fixes these issues. This is a rule we break it in certain rare instances but in generl keep line length to 80chars. if you use eclipse you can set general-editor-texteditor-showprintmargin, I find this useful single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-336: --- Assignee: Henry Robinson Status: Patch Available (was: Open) Hi Henry, from your comments it looks like you're ready for review, in ZooKeeper (hadoop in general) this is indicated by clicking submit patch on the left hand side of the JIRA page. This triggers the review/commit workflow - in particular it will run the hadoop qa bot and let us (committers) know to review for commit. The process is documented here: http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute It's also fine to submit drafts and ask for review, but I think in this case you are saying it's ready for final review (correct me if Im wrong). One nice thing about submitting is that it's listed on the project page as patch available and ensures that we won't overlook it. I will review this presently. single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Assignee: Henry Robinson Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch, ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-336) single bad client can cause server to stop accepting connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Robinson updated ZOOKEEPER-336: - Attachment: ZOOKEEPER-336.patch Attached is a patch which does the following: 1. Adds a configuration variable maxClientCnxns which defaults to 10, which limits the number of simultaneous connection attempts from an InetAddress. (This is the change that touches most files). 2. Update NIOServerCnxn.java to implement this change by closing a socket connection that contravenes this limit. I added a new constructor (called by the old one), plus the data structures to quickly find how many connections are already open from a particular address in O(1) (assuming finite maxClientCnxns) time. If maxClientCnxns is 0, there is no limit. Limitations: * If many clients are behind a NAT, this limit will have to be taken off as the code cannot distinguish between different NATted clients. * This probably isn't effective in the face of IPv6 if a single user has loads of addresses :) At present, clients that can't connect retry once every second. I also have a patch that adds a maxConnectionAttempt configuration variable, and have ClientCnxn enforce the limit, but I haven't found a clean way to call up to the ZooKeeper object and tell it to abandon the connection attempt (ZooKeeper.close() maybe seems to deadlock, I'll look further later). single bad client can cause server to stop accepting connections Key: ZOOKEEPER-336 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-336 Project: Zookeeper Issue Type: Improvement Components: c client, java client, server Reporter: Patrick Hunt Priority: Critical Fix For: 3.2.0 Attachments: ZOOKEEPER-336.patch One user saw a case where a single mis-programmed client was overloading the server with connections - the client was creating a huge number of sessions to the server. This caused all of the fds on the server to become used. Seems like we should have some way of limiting (configurable override) the maximum number of sessions from a single client (say 10 by default?) Also we should output warnings when this limit is exceeded (or attempt to exceed). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
