Re: Bad logins bogging down server
Follow up The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, might even explain the occasional slow response from IMAP I've observed.) I will run more experiments to see just how high the thread count should be, and I've got a list of other optimizations I will try. This note is in case somebody else sees the same problem. Mike On 09/16/2017 07:41 AM, Michael D. Sofka wrote: I'm seeking help from the collective wisdom of the Cyrus world. In the past two days we have seen first a doubling, and then a quadrupling+ of badlogins to Cyrus. These appear to be coming from a botnet, in that the IPs are spread around in a way that evades fail2ban. It got so bad Friday afternoon, that we took the extraordinary step of blocking off-campus connections to IMAP (email can still be read via Webmail and the VPN). The symptoms are that connections grow, and grow and grow until authentication slows, holding open connections longer and longer. It takes about 15 minutes for the connection number to be at a point at which service is interrupted. Friday night at attempt was made to re-enable off-campus IMAP, and the bots were still at it, service was again disrupted. But the number of connections does not appear close to max permitted by Cyrus. We have a Murder cluster: Three front-end servers, Two back-end servers, Two replication servers. The front-end servers are Ubuntu 14.04, Cyrus 2.4.17. The back-end and replication servers are Ubuntu 16.04, Cyrus 2.4.18. (Upgrading front-ends on the short list.) Authentication is via saslauthd, configured to use PAM, which is using krb5. Kerberos is running on three different kerberos servers. Load on the kerberos servers is light, and the kerb-admin says nowhere close to saturated. In fact, it handled much higher numbers of authentications before imapproxy on the Webmail service. (That was years ago, previous kerb servers, so there is still the possibility the kerberos servers are somehow slowed) Each Front-end server is configured for 5000 imapd on 143, and 5000 on port 993. Netstat shows about 4-5,000 imap connections per front-end server when authentication slows. There are well under 5000 imapd processes of either type. And after the Friday evening test re-allowing off-campus IMAP, the network admin reported about 1600 connections to port 993 total as IMAP authentication is slowed to a crawl. We are not close to file-max on any of the servers. imapd.conf has a 10 second delay for a badlogin. There are some mupdate log entries Thread timed out waiting for listener_lock Worker thread finished, for a total of 3 (2 spare) Around the time of the Friday afternoon problems, when I was restarting Front-end servers to recover. And no mupdate log entries since. What does this mean? There are entries in syslog when mupdate is restarted, stating that it could not reset the file limit to 5k. mupdate_connections_max is 1024, so the failure to reset has no affect, unless that is the limitation. But I see no log entries indicating that. Any other resources or limits in either Cyrus or Linux (Debian) that I should look at? Thank you in advance for any help. Mike -- Michael D. Sofka sof...@rpi.edu ITI Sr. Systems Programmer, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Using user_deny.db
We have many recalcitrant, bad, accounts constantly checking IMAP, long after the student has graduated. I would like to use user_deny.db to simply tell them to go away. First, would this offer an advantage? That is, does "login" check user_deny.db before authenticating, or after? Second, any examples of how to use cyr_dbtool (or other tool) to put entries into user_deny.db? Finally, my reading of the documentation (2.4.17/18) is that user_deny.db is a flat file by default, so I will need to set userdeny_db to something like skiplist, or berkeley, etc. Any suggestions on a good choice assuming the list could grow to a few thousand? Any documentation on the sql option? Thank You, Mike -- Michael D. Sofka sof...@rpi.edu ITI Sr. Systems Programmer, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Bad logins bogging down server
On 09/19/17 09:52 -0400, Michael Sofka wrote: The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, If you're comfortable with caching, increase the -t value to saslauthd. On 09/16/2017 07:41 AM, Michael D. Sofka wrote: The symptoms are that connections grow, and grow and grow until authentication slows, holding open connections longer and longer. It takes about 15 minutes for the connection number to be at a point at which service is interrupted. Friday night at attempt was made to re-enable off-campus IMAP, and the bots were still at it, service was again disrupted. Any other resources or limits in either Cyrus or Linux (Debian) that I should look at? https://debian-administration.org/article/187/Using_iptables_to_rate-limit_incoming_connections -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Using user_deny.db
On 09/19/17 10:02 -0400, Michael Sofka wrote: We have many recalcitrant, bad, accounts constantly checking IMAP, long after the student has graduated. I would like to use user_deny.db to simply tell them to go away. First, would this offer an advantage? That is, does "login" check user_deny.db before authenticating, or after? I believe that is it prior to authentication, based on my notes: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2010-June/033119.html Second, any examples of how to use cyr_dbtool (or other tool) to put entries into user_deny.db? Finally, my reading of the documentation (2.4.17/18) is that user_deny.db is a flat file by default, so I will need to set userdeny_db to something like skiplist, or berkeley, etc. Any suggestions on a good choice assuming the list could grow to a few thousand? Any documentation on the sql option? -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Using user_deny.db
On 09/19/2017 10:17 AM, Dan White wrote: On 09/19/17 10:02 -0400, Michael Sofka wrote: We have many recalcitrant, bad, accounts constantly checking IMAP, long after the student has graduated. I would like to use user_deny.db to simply tell them to go away. First, would this offer an advantage? That is, does "login" check user_deny.db before authenticating, or after? I believe that is it prior to authentication, based on my notes: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2010-June/033119.html user_deny.db is NOT checked prior to completion of LOGIN authentication, although it probably could/should. It works for POP3 USER/PASS because user_deny.db is checked in the command processing loop, so it happens between the USER and PASS commands. Second, any examples of how to use cyr_dbtool (or other tool) to put entries into user_deny.db? Finally, my reading of the documentation (2.4.17/18) is that user_deny.db is a flat file by default, so I will need to set userdeny_db to something like skiplist, or berkeley, etc. Any suggestions on a good choice assuming the list could grow to a few thousand? Any documentation on the sql option? -- Kenneth Murchison Cyrus Development Team FastMail Pty Ltd Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Bad logins bogging down server
On 09/19/2017 10:12 AM, Dan White wrote: The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, If you're comfortable with caching, increase the -t value to saslauthd. Interesting. What is the default value of -t? when just "-c" is specified? -- Michael D. Sofka sof...@rpi.edu ITI Sr. Systems Programmer, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Using user_deny.db
On 09/19/2017 10:28 AM, Ken Murchison wrote: I believe that is it prior to authentication, based on my notes: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2010-June/033119.html user_deny.db is NOT checked prior to completion of LOGIN authentication, although it probably could/should. It works for POP3 USER/PASS because user_deny.db is checked in the command processing loop, so it happens between the USER and PASS commands. Oh well. I agree that it would be a useful check before login authentication takes place. Meanwhile, any more comprehensive examples or documentation? Thank You, Mike -- Michael D. Sofka sof...@rpi.edu ITI Sr. Systems Programmer, Email, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Using user_deny.db
On 09/19/2017 11:31 AM, Michael Sofka wrote: On 09/19/2017 10:28 AM, Ken Murchison wrote: I believe that is it prior to authentication, based on my notes: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2010-June/033119.html user_deny.db is NOT checked prior to completion of LOGIN authentication, although it probably could/should. It works for POP3 USER/PASS because user_deny.db is checked in the command processing loop, so it happens between the USER and PASS commands. Oh well. I agree that it would be a useful check before login authentication takes place. There IS a check during the SASL proxy policy callback, but that isn't used for protocol-specific plaintext authentication commands. I just tested a quick patch which moved the check into the user canonicalization callback (which IS used my IMAP LOGIN, etc) and it works as expected. I would need to do further testing to make sure there aren't any unintended consequences. Meanwhile, any more comprehensive examples or documentation? https://www.cyrusimap.org/imap/concepts/deployment/databases.html#user-access-user-deny-db -- Kenneth Murchison Cyrus Development Team FastMail Pty Ltd Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Bad logins bogging down server
On 09/19/17 11:28 -0400, Michael Sofka wrote: On 09/19/2017 10:12 AM, Dan White wrote: The botnet is still hammering away, checking those old accounts. But the bottleneck appears to have been saslauthd threads. Doubling the thread count from 5 to 10 has resolved the problem for now. (And, If you're comfortable with caching, increase the -t value to saslauthd. Interesting. What is the default value of -t? when just "-c" is specified? It's much larger than I expected (from saslauthd/cache.h): #define CACHE_DEFAULT_TIMEOUT 28800 -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
