Re: Bad logins bogging down server
On 09/19/17 11:28 -0400, Michael Sofka wrote: On 09/19/2017 10:12 AM, Dan White wrote: The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, If you're comfortable with caching, increase the -t value to saslauthd. Interesting. What is the default value of -t? when just "-c" is specified? It's much larger than I expected (from saslauthd/cache.h): #define CACHE_DEFAULT_TIMEOUT 28800 -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Bad logins bogging down server
On 09/19/2017 10:12 AM, Dan White wrote: The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, If you're comfortable with caching, increase the -t value to saslauthd. Interesting. What is the default value of -t? when just "-c" is specified? -- Michael D. Sofka sof...@rpi.edu ITI Sr. Systems Programmer, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Bad logins bogging down server
On 09/19/17 09:52 -0400, Michael Sofka wrote: The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, If you're comfortable with caching, increase the -t value to saslauthd. On 09/16/2017 07:41 AM, Michael D. Sofka wrote: The symptoms are that connections grow, and grow and grow until authentication slows, holding open connections longer and longer. It takes about 15 minutes for the connection number to be at a point at which service is interrupted. Friday night at attempt was made to re-enable off-campus IMAP, and the bots were still at it, service was again disrupted. Any other resources or limits in either Cyrus or Linux (Debian) that I should look at? https://debian-administration.org/article/187/Using_iptables_to_rate-limit_incoming_connections -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Bad logins bogging down server
Follow up The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, might even explain the occasional slow response from IMAP I've observed.) I will run more experiments to see just how high the thread count should be, and I've got a list of other optimizations I will try. This note is in case somebody else sees the same problem. Mike On 09/16/2017 07:41 AM, Michael D. Sofka wrote: I'm seeking help from the collective wisdom of the Cyrus world. In the past two days we have seen first a doubling, and then a quadrupling+ of badlogins to Cyrus. These appear to be coming from a botnet, in that the IPs are spread around in a way that evades fail2ban. It got so bad Friday afternoon, that we took the extraordinary step of blocking off-campus connections to IMAP (email can still be read via Webmail and the VPN). The symptoms are that connections grow, and grow and grow until authentication slows, holding open connections longer and longer. It takes about 15 minutes for the connection number to be at a point at which service is interrupted. Friday night at attempt was made to re-enable off-campus IMAP, and the bots were still at it, service was again disrupted. But the number of connections does not appear close to max permitted by Cyrus. We have a Murder cluster: Three front-end servers, Two back-end servers, Two replication servers. The front-end servers are Ubuntu 14.04, Cyrus 2.4.17. The back-end and replication servers are Ubuntu 16.04, Cyrus 2.4.18. (Upgrading front-ends on the short list.) Authentication is via saslauthd, configured to use PAM, which is using krb5. Kerberos is running on three different kerberos servers. Load on the kerberos servers is light, and the kerb-admin says nowhere close to saturated. In fact, it handled much higher numbers of authentications before imapproxy on the Webmail service. (That was years ago, previous kerb servers, so there is still the possibility the kerberos servers are somehow slowed) Each Front-end server is configured for 5000 imapd on 143, and 5000 on port 993. Netstat shows about 4-5,000 imap connections per front-end server when authentication slows. There are well under 5000 imapd processes of either type. And after the Friday evening test re-allowing off-campus IMAP, the network admin reported about 1600 connections to port 993 total as IMAP authentication is slowed to a crawl. We are not close to file-max on any of the servers. imapd.conf has a 10 second delay for a badlogin. There are some mupdate log entries Thread timed out waiting for listener_lock Worker thread finished, for a total of 3 (2 spare) Around the time of the Friday afternoon problems, when I was restarting Front-end servers to recover. And no mupdate log entries since. What does this mean? There are entries in syslog when mupdate is restarted, stating that it could not reset the file limit to 5k. mupdate_connections_max is 1024, so the failure to reset has no affect, unless that is the limitation. But I see no log entries indicating that. Any other resources or limits in either Cyrus or Linux (Debian) that I should look at? Thank you in advance for any help. Mike -- Michael D. Sofka sof...@rpi.edu ITI Sr. Systems Programmer, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Bad logins bogging down server
I'm seeking help from the collective wisdom of the Cyrus world. In the past two days we have seen first a doubling, and then a quadrupling+ of badlogins to Cyrus. These appear to be coming from a botnet, in that the IPs are spread around in a way that evades fail2ban. It got so bad Friday afternoon, that we took the extraordinary step of blocking off-campus connections to IMAP (email can still be read via Webmail and the VPN). The symptoms are that connections grow, and grow and grow until authentication slows, holding open connections longer and longer. It takes about 15 minutes for the connection number to be at a point at which service is interrupted. Friday night at attempt was made to re-enable off-campus IMAP, and the bots were still at it, service was again disrupted. But the number of connections does not appear close to max permitted by Cyrus. We have a Murder cluster: Three front-end servers, Two back-end servers, Two replication servers. The front-end servers are Ubuntu 14.04, Cyrus 2.4.17. The back-end and replication servers are Ubuntu 16.04, Cyrus 2.4.18. (Upgrading front-ends on the short list.) Authentication is via saslauthd, configured to use PAM, which is using krb5. Kerberos is running on three different kerberos servers. Load on the kerberos servers is light, and the kerb-admin says nowhere close to saturated. In fact, it handled much higher numbers of authentications before imapproxy on the Webmail service. (That was years ago, previous kerb servers, so there is still the possibility the kerberos servers are somehow slowed) Each Front-end server is configured for 5000 imapd on 143, and 5000 on port 993. Netstat shows about 4-5,000 imap connections per front-end server when authentication slows. There are well under 5000 imapd processes of either type. And after the Friday evening test re-allowing off-campus IMAP, the network admin reported about 1600 connections to port 993 total as IMAP authentication is slowed to a crawl. We are not close to file-max on any of the servers. imapd.conf has a 10 second delay for a badlogin. There are some mupdate log entries Thread timed out waiting for listener_lock Worker thread finished, for a total of 3 (2 spare) Around the time of the Friday afternoon problems, when I was restarting Front-end servers to recover. And no mupdate log entries since. What does this mean? There are entries in syslog when mupdate is restarted, stating that it could not reset the file limit to 5k. mupdate_connections_max is 1024, so the failure to reset has no affect, unless that is the limitation. But I see no log entries indicating that. Any other resources or limits in either Cyrus or Linux (Debian) that I should look at? Thank you in advance for any help. Mike -- Michael D. Sofka sof...@rpi.edu C Sr. Systems Programmer, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
