[jira] [Commented] (QPID-4036) Failed client connections permanently exhaust broker's max connections limit
[ https://issues.apache.org/jira/browse/QPID-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416523#comment-13416523 ] Justin Ross commented on QPID-4036: --- Reviewed by Alan. Approved for 0.18. Corresponding commit is http://svn.apache.org/viewvc?view=revision&revision=1360214 . > Failed client connections permanently exhaust broker's max connections limit > > > Key: QPID-4036 > URL: https://issues.apache.org/jira/browse/QPID-4036 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: 0.16 > Environment: CentOS release 5.5 (Final) > Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 > x86_64 GNU/Linux > gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48) >Reporter: Paul Colby >Priority: Critical > Labels: sasl, ssl > > I'm running a set of Qpid 0.16 C++ brokers with configuration like: > {code} > cluster-name="mm-queue-cluster" > cluster-cman=yes > cluster-mechanism=PLAIN > cluster-username=broker > cluster-password=abc123 > cluster-url=ssl:gateway02:5671 > auth=yes > ssl-cert-db=/etc/qpid/certs/broker > ssl-cert-password-file=/etc/qpid/certs/pass.txt > ssl-cert-name=broker.messagemedia.com.au > require-encryption=yes > {code} > ie the broker is requiring both encryption and authentication (configured > SASL mech list is CRAM-MD5 DIGEST-MD5 EXTERNAL PLAIN). > Now, if a client (let's use {{qpid-stat}} for example) connects via SSL > (amqps) and authenticates successfully, then everything is happy. > However, if a client repeatedly fails to use SSL and/or fails to provide > credentials, then the broker loses one of it's configured max connections > every time! > So, for example, if we start the broker using the configuration shown above, > then do this: > {code}for i in `seq 1 550`; do echo $i; qpid-stat -q ; done{code} > The above loop will report ~ 500 {{AuthenticationFailure}} errors, then > switch to {{ConnectionError}} errors. Once the {{ConnectionError}} errors > begin, all further connections to the broker will be rejected - permanently > (until the broker is restarted), with the broker logging: > {code}error Client max connection count limit exceeded: 500 connection > refused{code} > From my testing, the following loops never cause an issue (with this > configuration): > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/guest@localhost > -q ; done # Works as expected. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/wrong@localhost > -q ; done # AuthenticationFailure as expected. > {code} > Whereas any of the following will break the broker: > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/guest@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/wrong@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-4036) Failed client connections permanently exhaust broker's max connections limit
[ https://issues.apache.org/jira/browse/QPID-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288146#comment-13288146 ] Paul Colby commented on QPID-4036: -- I've done a quick review of the changes from r1215127 that have since been dropped, and it seems unlikely that the subsequent changes would have regresses QPID-3438. Also, I've found a simpler test case: # Compile and install 0.16 from source; make sure CPG is enabled (ie install openais or corosync). # Configure SASL to allow only PLAIN (or at least disable ANONYMOUS). # Run: {{qpidd --cluster-name=test}} #* you'll probably need to run it as {{root}}, or some other user with permission to access the underlying cluster services. # Run: {{for i in `seq 1 550`; do echo $i; qpid-stat -q; done}} That's it... sit back and watch the broker become unusable ;) Incidentally, the problem does not occur if the broker is not instructed to use clustering. > Failed client connections permanently exhaust broker's max connections limit > > > Key: QPID-4036 > URL: https://issues.apache.org/jira/browse/QPID-4036 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: 0.16 > Environment: CentOS release 5.5 (Final) > Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 > x86_64 GNU/Linux > gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48) >Reporter: Paul Colby >Priority: Critical > Labels: sasl, ssl > > I'm running a set of Qpid 0.16 C++ brokers with configuration like: > {code} > cluster-name="mm-queue-cluster" > cluster-cman=yes > cluster-mechanism=PLAIN > cluster-username=broker > cluster-password=abc123 > cluster-url=ssl:gateway02:5671 > auth=yes > ssl-cert-db=/etc/qpid/certs/broker > ssl-cert-password-file=/etc/qpid/certs/pass.txt > ssl-cert-name=broker.messagemedia.com.au > require-encryption=yes > {code} > ie the broker is requiring both encryption and authentication (configured > SASL mech list is CRAM-MD5 DIGEST-MD5 EXTERNAL PLAIN). > Now, if a client (let's use {{qpid-stat}} for example) connects via SSL > (amqps) and authenticates successfully, then everything is happy. > However, if a client repeatedly fails to use SSL and/or fails to provide > credentials, then the broker loses one of it's configured max connections > every time! > So, for example, if we start the broker using the configuration shown above, > then do this: > {code}for i in `seq 1 550`; do echo $i; qpid-stat -q ; done{code} > The above loop will report ~ 500 {{AuthenticationFailure}} errors, then > switch to {{ConnectionError}} errors. Once the {{ConnectionError}} errors > begin, all further connections to the broker will be rejected - permanently > (until the broker is restarted), with the broker logging: > {code}error Client max connection count limit exceeded: 500 connection > refused{code} > From my testing, the following loops never cause an issue (with this > configuration): > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/guest@localhost > -q ; done # Works as expected. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/wrong@localhost > -q ; done # AuthenticationFailure as expected. > {code} > Whereas any of the following will break the broker: > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/guest@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/wrong@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-4036) Failed client connections permanently exhaust broker's max connections limit
[ https://issues.apache.org/jira/browse/QPID-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287816#comment-13287816 ] Paul Colby commented on QPID-4036: -- Thanks Andy. Yes QPID-3438 definitely looks like the same issue. However, as you said, that was supposedly fixed in 0.16, which I'm using. A quick survey shows that most (all?) of the commits for QPID-3438 (ie r1215127) are present in 0.16 as expected, however some of those changes have since been changed further (eg cluster/Connection.cppp Connection::connectionError). So, perhaps this issues is a regression against the fixes that were done for QPID-3438 (doesn't look likely), or another corner case that wasn't covered by QPID-3438 (more likely?). > Failed client connections permanently exhaust broker's max connections limit > > > Key: QPID-4036 > URL: https://issues.apache.org/jira/browse/QPID-4036 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: 0.16 > Environment: CentOS release 5.5 (Final) > Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 > x86_64 GNU/Linux > gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48) >Reporter: Paul Colby >Priority: Critical > Labels: sasl, ssl > > I'm running a set of Qpid 0.16 C++ brokers with configuration like: > {code} > cluster-name="mm-queue-cluster" > cluster-cman=yes > cluster-mechanism=PLAIN > cluster-username=broker > cluster-password=abc123 > cluster-url=ssl:gateway02:5671 > auth=yes > ssl-cert-db=/etc/qpid/certs/broker > ssl-cert-password-file=/etc/qpid/certs/pass.txt > ssl-cert-name=broker.messagemedia.com.au > require-encryption=yes > {code} > ie the broker is requiring both encryption and authentication (configured > SASL mech list is CRAM-MD5 DIGEST-MD5 EXTERNAL PLAIN). > Now, if a client (let's use {{qpid-stat}} for example) connects via SSL > (amqps) and authenticates successfully, then everything is happy. > However, if a client repeatedly fails to use SSL and/or fails to provide > credentials, then the broker loses one of it's configured max connections > every time! > So, for example, if we start the broker using the configuration shown above, > then do this: > {code}for i in `seq 1 550`; do echo $i; qpid-stat -q ; done{code} > The above loop will report ~ 500 {{AuthenticationFailure}} errors, then > switch to {{ConnectionError}} errors. Once the {{ConnectionError}} errors > begin, all further connections to the broker will be rejected - permanently > (until the broker is restarted), with the broker logging: > {code}error Client max connection count limit exceeded: 500 connection > refused{code} > From my testing, the following loops never cause an issue (with this > configuration): > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/guest@localhost > -q ; done # Works as expected. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/wrong@localhost > -q ; done # AuthenticationFailure as expected. > {code} > Whereas any of the following will break the broker: > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/guest@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/wrong@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-4036) Failed client connections permanently exhaust broker's max connections limit
[ https://issues.apache.org/jira/browse/QPID-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287405#comment-13287405 ] Andy Goldstein commented on QPID-4036: -- This sounds similar to QPID-3438, which was supposed to be fixed in 0.16... > Failed client connections permanently exhaust broker's max connections limit > > > Key: QPID-4036 > URL: https://issues.apache.org/jira/browse/QPID-4036 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: 0.16 > Environment: CentOS release 5.5 (Final) > Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 > x86_64 GNU/Linux > gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48) >Reporter: Paul Colby >Priority: Critical > Labels: sasl, ssl > > I'm running a set of Qpid 0.16 C++ brokers with configuration like: > {code} > cluster-name="mm-queue-cluster" > cluster-cman=yes > cluster-mechanism=PLAIN > cluster-username=broker > cluster-password=abc123 > cluster-url=ssl:gateway02:5671 > auth=yes > ssl-cert-db=/etc/qpid/certs/broker > ssl-cert-password-file=/etc/qpid/certs/pass.txt > ssl-cert-name=broker.messagemedia.com.au > require-encryption=yes > {code} > ie the broker is requiring both encryption and authentication (configured > SASL mech list is CRAM-MD5 DIGEST-MD5 EXTERNAL PLAIN). > Now, if a client (let's use {{qpid-stat}} for example) connects via SSL > (amqps) and authenticates successfully, then everything is happy. > However, if a client repeatedly fails to use SSL and/or fails to provide > credentials, then the broker loses one of it's configured max connections > every time! > So, for example, if we start the broker using the configuration shown above, > then do this: > {code}for i in `seq 1 550`; do echo $i; qpid-stat -q ; done{code} > The above loop will report ~ 500 {{AuthenticationFailure}} errors, then > switch to {{ConnectionError}} errors. Once the {{ConnectionError}} errors > begin, all further connections to the broker will be rejected - permanently > (until the broker is restarted), with the broker logging: > {code}error Client max connection count limit exceeded: 500 connection > refused{code} > From my testing, the following loops never cause an issue (with this > configuration): > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/guest@localhost > -q ; done # Works as expected. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/wrong@localhost > -q ; done # AuthenticationFailure as expected. > {code} > Whereas any of the following will break the broker: > {code:none} > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/guest@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/wrong@localhost > -q ; done # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://localhost -q ; done > # AuthenticationFailure, then ConnectionError. > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org