Hi Eric - was that for Andy or me I'm on
- qmail-1.03-3.1.1.qt.el7.x86_64 - qmailadmin-1.2.16-2.qt.el7.x86_64 - qmailmrtg-4.2-3.qt.el7.x86_64 David Bray 0418 745334 2 ∞ & < On Tue, 21 Apr 2020 at 23:34, Eric Broch <ebr...@whitehorsetc.com> wrote: > Andy, > > May I ask what version of qmail you're on? > > Eric > On 4/20/2020 10:15 PM, David Bray wrote: > > Hi Andy - you have grasped the problem accurately > > As I understand it, the transaction does not leave a great deal of scope - > negotiate the encryption, send a password successfully or unsuccessfully - > (at that point it's logged) > > So it has to be the negotiation phase ... > but: > > - I've only just built this server > - stuck to a standard install using a CentOS 7 VM - 4GB: 2 CPU, 80GB > - I think this adequate I've seen no OOM events - and the size is > what I've used before > > The only thing I'm really seeing here that could be an issue is - the > newer machines are stricter on SSL - the TLS 1/1.2 deprecation thing > I see lots of broken servers, and have to *make allowances*, I do this by: > > touch /var/qmail/control/notlshosts/<the host name> > > > Noting - that is an outbound thing ... (see > https://www.qmailtoaster.org/notls.html) > > So ...... is it possible that a broken client is hitting the port, not > being able to make the necessary handshake and causing this CPU load .... > It's reported in the logs when the server makes an outbound transaction > like that ... > > deferral: > TLS_connect_failed:_error:14077410:SSL_routines:SSL23_GET_SERVER_HELLO:sslv3_alert_handshake_failure;_connected_to_103.27.32.20./ > > > What would it look like in my logs if they where to have the reverse issue > > > > > > David Bray > 0418 745334 > 2 ∞ & < > > > On Tue, 21 Apr 2020 at 02:54, Andrew Swartz <awswa...@acsalaska.net> > wrote: > >> Port 465 should be SMTP over SSL/TLS. Therefore the sequence of events >> is: >> >> 1. Establish TCP connection >> 2. Negotiate SSL/TLS session >> 3. Begin SMTP session >> >> Of these, the SSL/TLS negotiation is by far the most CPU-intensive. >> >> Consider trying to see what is happening with the SSL/TLS negotiation. >> It may be failing in a way which is slamming the CPU but not showing up >> in the SMTP logs because it never completes and thus an SMTP session is >> never initiated. >> >> I'm unsure the best way to debug the incoming SSL/TLS negotiation. You >> might set a firewall rule where incoming port 465 from that single IP is >> forwarded to stunnel (on another machine) which is set to output verbose >> debug info??? >> >> It would be interesting to know the cause. This could be some clever >> DOS attack where a single connection accomplishes the DOS by slamming >> the CPU by submitting something invalid to openSSL. But it might just >> be that some spammer is using a home-brewed script which is buggy and is >> unintentionally causing this problem. >> >> There seems no efficient way to block this without figuring out the >> cause and doing something to make that cause be logged into some log >> file. Once that is accomplished, fail2ban (or something similar) can >> easily add firewall rules to block individual IP's which exhibit this >> behavior. >> >> -Andy >> >> >> >> >> >> On 4/19/2020 10:12 PM, David Bray wrote: >> > Hey thanks Remo >> > smtps is an inbound port, they are contacting me - this IP is in Russia >> > somewhere - so do I want to engage (perhaps, probably not but ..) >> > >> > I could of course block that IP - but that doesn't help, I'd have to >> > block endless IPs >> > >> > I'd like to know what's taking the CPU load, in theory they should be >> > connecting, supplying a password (perhaps) and sending data >> > but, there are sometimes bad passwords (2 for the 20th recorded in >> maillog) >> > >> > So.. >> > What are they doing the other times and why is it taking so much CPU - >> > if it is just a port knock, why the CPU >> > >> > I need to be able to say, they are bad because ... *what is the >> because* ? >> > >> > David Bray >> > 0418 745334 >> > 2 ∞ & < >> > >> > >> > On Mon, 20 Apr 2020 at 15:32, Remo Mattei <r...@mattei.org >> > <mailto:r...@mattei.org>> wrote: >> > >> > Hi, >> > Can you reach the server? It maybe blocking you. So what does your >> > queue looks like? >> > >> > Here is mine for example: >> > >> > # qmHandle -L >> > Messages in local queue: 0 >> > Messages in remote queue: 0 >> > >> > My other server >> > >> > # qmHandle -L >> > 10355792 (19, L) >> > Return-path: r...@qmailxxxxx.com <mailto:r...@qmailxxxxx.com> >> > From: Anacron <r...@qmailxxxx.com <mailto:r...@qmailxxxx.com>> >> > To: r...@qmailxxxxx.com <mailto:r...@qmailxxxxx.com> >> > Subject: Anacron job 'cron.daily' on qmailxxxxx.com >> > <http://qmailxxxxx.com> >> > Date: 19 Apr 2020 10:28:28 -0000 >> > Size: 509 bytes >> > >> > 10358746 (6, L) >> > Return-path: >> > From: mailer-dae...@qmailxxxxxx.com >> > <mailto:mailer-dae...@qmailxxxxxx.com> >> > To: r...@qmailxxxxxxxx.com <mailto:r...@qmailxxxxxxxx.com> >> > Subject: failure notice >> > Date: 19 Apr 2020 11:30:30 -0000 >> > Size: 1089 bytes >> > >> > Messages in local queue: 2 >> > Messages in remote queue: 0 >> > >> > Just wonder it looks that you are using the SMTPs 465, did you try >> > the 587 Submission that address and see if it goes? >> > Just wonder if this is tight to that service. >> > >> > Maybe none of the above but just for troubleshooting steps, I would >> > try that. >> > >> > Remo >> > >> > >> >> On Apr 19, 2020, at 22:11, David Bray <da...@brayworth.com.au >> >> <mailto:da...@brayworth.com.au>> wrote: >> >> >> >> Ok - but after all the investigation etc, this is actually the >> >> trigger which caught my eye in the first place >> >> >> >> How this comes about is: >> >> >> >> 1. User say hey I can't send >> >> 2. I look and see this high CPU load and intermittent failures >> >> for client to send >> >> >> >> Any thoughts on where to start looking .. >> >> >> >> >> >> <image.png> >> >> >> >> my smtp/smtps are currently *10*/11 connections >> >> >> >> >> >> ==> /var/log/qmail/smtp/current <== >> >> 2020-04-20 05:07:50.207299500 tcpserver: end 29699 status 0 >> >> 2020-04-20 05:07:50.207300500 tcpserver: status: 0/60 >> >> >> >> ==> /var/log/qmail/smtps/current <== >> >> 2020-04-20 05:07:54.903665500 tcpserver: status: 9/60 >> >> 2020-04-20 05:07:54.936654500 tcpserver: pid 29725 from >> 185.50.149.5 >> >> 2020-04-20 05:07:54.936655500 tcpserver: ok 29725 >> >> dev.brayworth.com <http://dev.brayworth.com>:172.105.181.18:465 >> >> <http://172.105.181.18:465> :185.50.149.5::5622 >> >> 2020-04-20 05:08:00.108657500 tcpserver: status: 10/60 >> >> 2020-04-20 05:08:00.152909500 tcpserver: pid 29734 from >> 185.50.149.5 >> >> 2020-04-20 05:08:00.152910500 tcpserver: ok 29734 >> >> dev.brayworth.com <http://dev.brayworth.com>:172.105.181.18:465 >> >> <http://172.105.181.18:465> :185.50.149.5::62006 >> >> 2020-04-20 05:08:05.172650500 tcpserver: status: *11*/60 >> >> 2020-04-20 05:08:05.208983500 tcpserver: pid 29740 from >> 185.50.149.5 >> >> 2020-04-20 05:08:05.208984500 tcpserver: ok 29740 >> >> dev.brayworth.com <http://dev.brayworth.com>:172.105.181.18:465 >> >> <http://172.105.181.18:465> :185.50.149.5::19686 >> >> 2020-04-20 05:08:13.601336500 tcpserver: end 29690 status 256 >> >> 2020-04-20 05:08:13.601337500 tcpserver: status: 10/60 >> >> >> >> David Bray >> >> 0418 745334 >> >> 2 ∞ & < >> >> >> >> >> >> On Sun, 19 Apr 2020 at 10:04, David Bray <da...@brayworth.com.au >> >> <mailto:da...@brayworth.com.au>> wrote: >> >> >> >> Thanks Eric >> >> >> >> It's hard to track things but I think I have had success >> >> monitoring the /var/log/maillog >> >> >> >> I'm not sure why I didn't pick this up earlier, I'm >> >> already using the fail2ban suggestion of the older >> >> qmailtoaster wiki >> >> (http://wiki.qmailtoaster.com/index.php/Fail2Ban), actually >> >> had a rule to process it and have expanded on this now >> >> >> >> I've been running email servers most of my working life and >> >> still get tripped up by simple stuff >> >> >> >> Thank for your efforts in this area, it helps to talk things >> out >> >> >> >> cheers >> >> >> >> David Bray >> >> 0418 745334 >> >> 2 ∞ & < >> >> >> >> >> >> On Sun, 19 Apr 2020 at 01:12, Eric Broch >> >> <ebr...@whitehorsetc.com <mailto:ebr...@whitehorsetc.com>> >> wrote: >> >> >> >> It looks like a connect and disconnect. If there was >> >> authentication you'd see it. I don't think you have >> >> anything to worry about here. I'm not saying there's not >> >> some jerk out there messing with your smtps...just saying >> >> it may be harmless. That said, do you have a good firewall >> >> in place that prevents DOS attacks. I use Sonicwall myself >> >> but you can do the same thing as others have shown with >> >> iptables. >> >> >> >> Does anyone know how to do the same with the stock >> >> firewalld on COS7/8? >> >> >> >> On 4/17/2020 11:49 PM, David Bray wrote: >> >>> sure - thanks for replying, this comes in waves taking >> >>> the server to it's maximum at times >> >>> >> >>> as far as I can see this only logs are this: >> >>> >> >>> ==> /var/log/qmail/smtps/current <== >> >>> 2020-04-18 05:04:48.450871500 tcpserver: status: 6/60 >> >>> 2020-04-18 05:04:48.480785500 tcpserver: pid 13339 from >> >>> 141.98.80.30 >> >>> 2020-04-18 05:04:48.480787500 tcpserver: ok 13339 >> >>> dev.brayworth.com >> >>> <http://dev.brayworth.com>:172.105.181.18:465 >> >>> <http://172.105.181.18:465> :141.98.80.30::25638 >> >>> 2020-04-18 05:04:52.797644500 tcpserver: status: 7/60 >> >>> 2020-04-18 05:04:52.830767500 tcpserver: pid 13340 from >> >>> 141.98.80.30 >> >>> 2020-04-18 05:04:52.830768500 tcpserver: ok 13340 >> >>> dev.brayworth.com >> >>> <http://dev.brayworth.com>:172.105.181.18:465 >> >>> <http://172.105.181.18:465> :141.98.80.30::14862 >> >>> 2020-04-18 05:04:57.248902500 tcpserver: status: 8/60 >> >>> 2020-04-18 05:04:57.304003500 tcpserver: pid 13342 from >> >>> 141.98.80.30 >> >>> 2020-04-18 05:04:57.304006500 tcpserver: ok 13342 >> >>> dev.brayworth.com >> >>> <http://dev.brayworth.com>:172.105.181.18:465 >> >>> <http://172.105.181.18:465> :141.98.80.30::9646 >> >>> 2020-04-18 05:05:01.854790500 tcpserver: status: 9/60 >> >>> 2020-04-18 05:05:01.902265500 tcpserver: pid 13345 from >> >>> 141.98.80.30 >> >>> 2020-04-18 05:05:01.902266500 tcpserver: ok 13345 >> >>> dev.brayworth.com >> >>> <http://dev.brayworth.com>:172.105.181.18:465 >> >>> <http://172.105.181.18:465> :141.98.80.30::54058 >> >>> 2020-04-18 05:05:09.729711500 tcpserver: end 13338 status >> 256 >> >>> 2020-04-18 05:05:09.729713500 tcpserver: status: 8/60 >> >>> 2020-04-18 05:06:05.965715500 tcpserver: end 13342 status >> 256 >> >>> 2020-04-18 05:06:05.965716500 tcpserver: status: 7/60 >> >>> 2020-04-18 05:06:06.141272500 tcpserver: end 13340 status >> 256 >> >>> 2020-04-18 05:06:06.141273500 tcpserver: status: 6/60 >> >>> >> >>> David Bray >> >>> 0418 745334 >> >>> 2 ∞ & < >> >>> >> >>> >> >>> On Sat, 18 Apr 2020 at 15:41, Eric Broch >> >>> <ebr...@whitehorsetc.com >> >>> <mailto:ebr...@whitehorsetc.com>> wrote: >> >>> >> >>> Can you send the log of one of the "bad" connections? >> >>> >> >>> On 4/17/2020 10:59 PM, David Bray wrote: >> >>> >> >>>> I can see I'm getting hammered on my smtps port >> >>>> >> >>>> How can I mitigate this? >> >>>> >> >>>> I can see the IP's in /var/log/qmail/smtps/current >> >>>> >> >>>> *but where do I actually see that the smtp auth >> >>>> actually fails ?* >> >>>> >> >>>> or do I need to increase the logging somewhere ? >> >>>> >> >>>> if I tail -f /var/log/dovecot.log >> >>>> >> >>>> I can see the imap and pop failures >> >>>> >> >>>> thanks in advance >> >>>> >> >>>> David Bray >> >>>> 0418 745334 >> >>>> 2 ∞ & < >> >>> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com >> For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com >> >>
