Mixed results on this. So far no problems with running workers being logged but the GUI has become incredibly unresponsive. By unresponsive I mean I waited a good couple of minutes for the shutdown_list page to load. The dot on the main page is red yet the workers page is all green. Scratch that, it has refreshed again and I have a worker stuck: Worker 3, loop age 252s, action: header (Content-Disposition -attr) : : filename name (stuck) 30s later and it is healthy again..
On the server I haven't upgraded the shutdown_list page comes up within seconds. I'm not sure whether to leave it running or whether this is evidence of the same kind of unresponsiveness that cause me to have to roll back earlier this week. On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote: > I wish I'd spotted this before writing out the other message. I'll give it > a test now for you. > > On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt < > thomas.ecka...@thockar.com> wrote: > >> Collin, this should no longer happen using the updated 2.5.2 16274_1 at >> CVS /test >> >> Thomas >> >> >> >> Von: cw <colin.war...@gmail.com> >> An: ASSP development mailing list <assp-test@lists.sourceforge.net> >> Datum: 29.09.2016 16:40 >> Betreff: Re: [Assp-test] Inbound TLS from gmail.com addresses / >> servers >> >> >> >> Hi Thomas, >> I moved up to 16270 following this thread of discussion but then had a day >> working away. I've come back to huge issues with delays, mails not going >> through and many, many of these in the logs: >> >> Info: unable to detect any running worker for a new connection - wait (max >> 30 seconds) >> >> When I say many, I have over 21,000 lines in today's log file. I also >> found >> the GUI unresponsive or not connecting at all and ASSP restarting quite >> regularly. >> >> I've dropped back to 16256 and things are instantly better. Do you think >> going up to 16273 might improve things over 16270 or am I better holding >> off for now? >> All the best, >> Colin. >> >> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt >> <thomas.ecka...@thockar.com> >> wrote: >> >> > I just released 2.5.2 build 16273 at CVS test folder >> > >> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/ >> > >> > This release should make a very large difference for SSL/TLS mails sent >> by >> > hosts that uses small SSL-frame size. >> > >> > Tell me your test results. >> > >> > >> > Thomas >> > >> > >> > >> > >> > >> > Von: K Post <nntp.p...@gmail.com> >> > An: ASSP development mailing list <assp-test@lists.sourceforge.net> >> > Datum: 28.09.2016 19:42 >> > Betreff: Re: [Assp-test] Inbound TLS from gmail.com addresses / >> > servers >> > >> > >> > >> > But I want a postman driving a Ferarri with monster truck tires that can >> > roll over the traffic (and if wishes are being granted, I'd prefer the >> car >> > in a deep blue instead of classic red). >> > >> > We regularly see people attaching large files or a bunch of smaller ones >> > that add up to a big email, I'm talking lots and lots of different >> people >> > from outside the organization sending to us, and this happens on a daily >> > basis. It's especially popular with photos and huge scans multi-page >> > 600dpi (which people don't understand can be done at low resolution). >> > Often it's people sending in scanned official documents for us to review >> > an >> > help them. They're not our staff, they're the people we help. They >> have >> > a >> > tendency of not following any instructions, and ignore the fact that we >> > have a web based system for the process. We can't control it and the >> > powers that be don't want us lowering the 30 MB threshold across the >> > board. Lot of these people use gmail.com addresses and google allows >> for >> > up to 25 MB - https://support.google.com/mail/answer/6584 >> > >> > I think it's really interesting that google seems to use this >> inefficient >> > small packet size for SSL, allows for 25MB emails, is a big proponent of >> > SSL, and at the same time doesn't allow mails to take more than 15 >> minutes >> > to transfer. Now that you've made things >much< more efficient on the >> > ASSP >> > side, I'm hoping that all will be okay. I just get annoyed by >> > inefficiency. >> > >> > >> > I'm going to tryrunning with npSize of zero, the no queuing size set >> very >> > high and see how that goes. I want to insure that even the biggest >> emails >> > are scanned for malware before hitting the inbox. >> > >> > >> > I've never said this before, but it seems like google's doing it wrong. >> I >> > think we've exhausted our options here. If anyone has a Google >> > engineering >> > friend, or a friend of a friend, this might be a good time to have a >> > little >> > chat in an effort to reach someone who can either explain why they're >> use >> > a >> > 1.4 kB frame size for ssl packets, but a normal much bigger size for >> > non-encrypted traffic or to get them to see an error in their ways and >> fix >> > this! >> > >> > Thank you again for all of the discussion and explanation and of course >> > the >> > code changes which have made a huge difference. >> > >> > On Wed, Sep 28, 2016 at 12:59 PM, Thomas Eckardt >> > <thomas.ecka...@thockar.com >> > > wrote: >> > >> > > >I'm still afraid of running into frequent problems with 25mb >> > attachments >> > > though. >> > > >> > > 25MB - I don't know anyone who allows this. Sending a link to a >> download >> > > is much smarter. >> > > ASSP_AFC has an option to do this for your outgoing mails! - >> > > ASSP_AFCWebScript >> > > >> > > >Can someone else run a debug test with Gmail and TLS to see if you're >> > > seeing these tiny packets too? >> > > >> > > I've done it - same behavior - 1400 byte per SSL frame from gmail. >> > > >> > > >> > > >Would it be an easy modification to show the negotiated cipher in the >> > > >> > > ASSP does this for years now - simply look in to the received header >> > line >> > > added by assp. >> > > >> > > -- -- -- Received: from vs10241.internet1.de ([158.181.49.77] >> > > helo=vs10241.internet1.de) >> > > by xxxxxx with SMTPS(TLSv1_2 ECDHE-RSA-AES256-GCM-SHA384) >> > (2.5.2); >> > > 24 Jul 2016 05:34:12 +0200 >> > > >> > > if you set 'ConnectionLog' at least to verbose, you'll see the >> > > SSL-negotiation results in maillog.txt. >> > > >> > > info: started TLS-SSL session for client $cliIP - using >> > > $Con{$ssls}->{sslversion} , $Con{$ssls}->{sslcipher} >> > > >> > > >what other options are there? >> > > >> > > Set 'EnableHighPerformance' to very high. >> > > >> > > >Are you sure that negotiating a lesser cipher with Gmail wouldn't >> have >> > > them switch to sending larger SSL packets? >> > > >> > > Yes, I'm 100% sure. >> > > >> > > An SSL peer is free to use any SSL frame size between 1 byte and 16kB. >> > > There is no SSL-negotiation parameter for min or max frame size. >> > > If gmail.com sends 1.4kB frames, they are free to do it this way. >> There >> > is >> > > no technical way to force them to send larger frames. >> > > >> > > >If neverQueue is set to 12MB, am I correct in saying that we're open >> to >> > a >> > > targeted exploit, say an .exe in a .zip as long as the email is more >> > than >> > > 12MB? >> > > >> > > Yes, you are correct. >> > > >> > > >or is that just after the whole message is received? >> > > >> > > Yes, after the complete mail is received. >> > > >> > > It is like it is - if the postman with his Ferrari is in a traffic >> jam, >> > > you'll have to wait longer for your letter or you'll get it the next >> > day. >> > > >> > > >> > > Thomas >> > > >> > > >> > > >> > > Von: K Post <nntp.p...@gmail.com> >> > > An: ASSP development mailing list >> <assp-test@lists.sourceforge.net> >> > > Datum: 28.09.2016 17:03 >> > > Betreff: Re: [Assp-test] Inbound TLS from gmail.com addresses >> / >> > > servers >> > > >> > > >> > > >> > > Thanks again for the detailed explanation. >> > > >> > > 10% faster is terrific, and it's more than 50% faster since before >> > 16270. >> > > I'm still afraid of running into frequent problems with 25mb >> attachments >> > > though. Yes, that's too big for email in our opinion, but Gmail >> allows >> > it >> > > as do most ISP's these days. It's not terribly unusual for us to have >> a >> > a >> > > person try to send and email with four or five 4 MB photos - and we >> > can't >> > > stop that trend. That'll translate into a message somewhere in the >> > range >> > > of 22-25 MB. I'm not sure if we're fast enough to >consistently< >> > receive >> > > these. And on top of that, if it's an outside big wig, emailing an >> > inside >> > > big wig, they don't have 15 minutes to waste waiting for the big email >> > to >> > > arrive. Then they yell at me, which is why I keep asking more of >> these >> > > questions -- and there's a lot of them here (sorry) >> > > >> > > >> > > *Are you sure that negotiating a lesser cipher with Gmail wouldn't >> have >> > > them switch to sending larger SSL packets? * I have a solid high level >> > > understanding of encryption, but once you get down the the packet >> level, >> > > I'm a bit out of my depth, despite what I've learned from your >> excellent >> > > explanations. >> > > >> > > >> > > Can someone else run a debug test with Gmail and TLS to see if you're >> > > seeing these tiny packets too? I keep going back to *not being able >> to >> > > believe that Google doing something that was any slower than >> necessary. >> > * >> > > They >> > > try to optimize everything and are huge proponents of SSL everywhere. >> If >> > > they're sending tons of tiny un-optimized packets, there's got to be a >> > > reason or cause. I can't imagine that it's a default and deliberate >> > > setting. I tried calling US support, but this is a network >> engineering >> > > type of issue - no chance of me getting through to someone there who >> > even >> > > understands the question. >> > > >> > > The npSize option and neverQueueSize, only affects things after the >> > whole >> > > message is received right? * It's not like it's doing something >> > > significantly extra at each SSL cylce is it except using up RAM >> right??* >> > > My unscientific tests are shower marginally slower speeds when >> > receiving >> > > large email from Google with TLS on and neverQueueSize set crazy high, >> > so >> > > maybe I'm wrong. Being that this same message through outlook.com >> only >> > > takes 30 seconds, even with npSize set to 0 (unlimited) and >> > neverQueueSize >> > > = 999,000,000, I'm wondering if the processing power and ram >> > availability >> > > (24 GB) on my assp box combined with low usage (only averaging 5000 >> > > messages a day) might be enough to have neverQueueSize really high to >> > > always scan everything. I'll play around there, but if neverQueueSize >> > is >> > > 99MB, will things like ASSP_AFCDetectSpamAttachRe be slow WHILE the >> > > message >> > > is transferring, or is that just after the whole message is received? >> > > >> > > If neverQueue is set to 12MB, am I correct in saying that we're open >> to >> > a >> > > targeted exploit, say an .exe in a .zip as long as the email is more >> > than >> > > 12MB? If so, that seems like a big hole, and something that I >> > personally >> > > would be willing to sacrifice speed/processing power to close. >> > > >> > > Suggestion, note in the gui under npSize that 0 really sets this to a >> > > maximum of 12,000,000 unless neverQueueSize is set higher in >> > > CorrectASSPcfg.pm. I've got that correct, right?) >> > > >> > > I forgot that DKIM signing can be done over the body too! We don't do >> > > that, but I forgot that others might. Maybe we have npSize only >> ignore >> > > DKIM if it's done over the body? DKIM should be fast if it's just the >> > > typical headers that are used right? We have npSize set to zero >> > though, >> > > so this doesn't really impact me. >> > > >> > > Would it be an easy modification to show the negotiated cipher in the >> > > maillog or in the email body, if nothing else, just to have more info? >> > > X-ASSP-Cipher: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384 >> > > for example or in the received line like Microsoft does: >> > > >> > > Received: from __________ (x.x.x.x) by ______ >> > > (y.y.y.y) with Microsoft SMTP Server (version=TLS1_2, >> > > cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) >> > > >> > > Or like Gmail does >> > > Received: from smtp.ourcharity.org (smtp.ourcharity.org. [x.x.x.x]) >> > > by mx.google.com with ESMTPS id >> > > l1si54144332.11.2016.09.28.07.51.21 >> > > for <nntp.p...@gmail.com> >> > > (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 >> > bits=128/128); >> > > Wed, 28 Sep 2016 07:24:22 -0700 (PDT) >> > > >> > > It's a little more data (and certainly your time to code this), but >> > could >> > > be useful if we need it in the future. >> > > >> > > and:: >> > > >> > > Again Ken, the SSL parameters are NOT the problem on your system. Your >> > > >> > > debug log shows a socket read time of max ~ 0.5 milliseconds (typical >> > ~0.3 >> > > >> > > ms) for SSL. This read operation includes the time required for the >> > > >> > > decryption of the data. This is very very fast! >> > > >> > > It is simply the amazing count (10.400) of read and process operations >> > > >> > > (cycles) required by assp for such a mail, that causes the overall >> slow >> > > >> > > mail receive. >> > > >> > > in summary, if there's not an ASSP setting that I can tweak to have >> > google >> > > start sending larger SSL packets like Outlook.com and everyone else >> I've >> > > seen does, what other options are there? Are you suggesting that ALL >> > ASSP >> > > installations are incapable of receiving large TLS emails from Google >> > > regardless of the installation's processing power, RAM, and bandwidth? >> > > >> > > >> > > >> > > On Wed, Sep 28, 2016 at 4:26 AM, Thomas Eckardt >> > > <thomas.ecka...@thockar.com> >> > > wrote: >> > > >> > > > >The same email now took only 269 seconds. >> > > > >> > > > This is OK for googles behavior - the 1440 byte SSL frame. Around >> 300s >> > > was >> > > > expected by me for this mail - hmm.. it is 10% better - nice! >> > > > >> > > > Take the following math. >> > > > >> > > > mail size = 15000 kB >> > > > google frame size = 1.44 kB >> > > > >> > > > required assp loop cycles = mail size / frame size = 10.400 >> > > > >> > > > mail size = 15000 kB >> > > > outlook.com frame size = 16 kB >> > > > >> > > > required assp loop cycles = mail size / frame size = 940 >> > > > >> > > > >I really wonder if I need to tweak my cipher_list. >> > > > >> > > > No! >> > > > Again Ken, the SSL parameters are NOT the problem on your system. >> Your >> > > > debug log shows a socket read time of max ~ 0.5 milliseconds >> (typical >> > > ~0.3 >> > > > ms) for SSL. This read operation includes the time required for the >> > > > decryption of the data. This is very very fast! >> > > > It is simply the amazing count (10.400) of read and process >> operations >> > > > (cycles) required by assp for such a mail, that causes the overall >> > slow >> > > > mail receive. >> > > > >> > > > >Is ClamAV scanning skipped too? >> > > > Yes. >> > > > >> > > > >Could the plugins be run on the full mail after receipt, regardless >> > of >> > > > size? >> > > > >> > > > Override the config parameters. Keep in mind that 'npSize' may also >> > > > involved in skipping or processing some mail body checks. >> > > > >> > > > >Isn't DKIM checking just a one time thing and not intensive? >> > > > >> > > > The full DKIM check is very intensive. It requires to calculate an >> > > RSA/SHA >> > > > hash over the complete mail. >> > > > DCC and Razor are doing something similar. >> > > > ASSP_AFC would make all checks (ClamAV, FileScan, content checks >> with >> > > > several regular expression, decompression of attachments ....) for >> the >> > > > complete mail. It parses the complete mail at once with Email::MIME. >> > > This >> > > > requires a huge amount of memory. Not a big deal on a 64bit OS with >> > 8GB >> > > > RAM and several CPU cores - who can !? >> > > > All these can be done for any mail size, if the system is able to >> > > process >> > > > the amount of data fast enough. >> > > > On most havy load systems, the 12.000.000 will be to large and may >> > lead >> > > in >> > > > to stucking workers. >> > > > >> > > > ASSP has this set of config parameters - change them to your need - >> > try >> > > - >> > > > and if does not work switch back - nothing more easy. >> > > > >> > > > Thomas >> > > > >> > > > >> > > > >> > > > Von: K Post <nntp.p...@gmail.com> >> > > > An: ASSP development mailing list >> > <assp-test@lists.sourceforge.net> >> > > > Datum: 27.09.2016 21:08 >> > > > Betreff: Re: [Assp-test] Inbound TLS from gmail.com >> addresses >> / >> > > > servers >> > > > >> > > > >> > > > >> > > > Our primary internet connection went down again (nothing to do with >> > > ASSP) >> > > > which gave me the opportunity to replace 16270 with 16271. Nothing >> > like >> > > > making lemonade out of lemons... >> > > > >> > > > The same email now took only 269 seconds. That's about 15x longer >> > than >> > > > with TLS off, but WOW that's way better than it was before. >> > > > I also tried with the blank cipher list, no notable difference >> > > > And with the SSL buffers set to 0 (64 MB), again without a speed >> > > > difference. >> > > > >> > > > SO- you've made a real difference here!! Is there more >> optimization >> > to >> > > > be >> > > > made? >> > > > >> > > > The rub is that the exact same message sent through Outlook.com to >> us, >> > > > took >> > > > exactly 30 seconds, just a 50% overhead when compared to the 19 >> > seconds >> > > > for >> > > > a non-TLS message of the same size, instead of a 1500% overhead for >> > > > encryption when receiving from google. >> > > > >> > > > *Is there some magical debug switch that I could turn on to see what >> > > > encryption Outlook.com is using and compare that to what Google's >> > > > connection to us with?* I think prohibiting whatever the slow >> cipher >> > > that >> > > > google's using (probably a really strong one) might make the final >> bit >> > > of >> > > > difference. >> > > > >> > > > I'm breathing so much easier now!! Thank you. >> > > > >> > > > >> > > > On Tue, Sep 27, 2016 at 2:08 PM, K Post <nntp.p...@gmail.com> >> wrote: >> > > > >> > > > > Despite all the problems we have with personalities and policies >> in >> > > our >> > > > > organization, the infrastructure is pretty solid. MTU's are set >> > > > correctly, >> > > > > no fragmentation, no jitter. There's low latency across the >> board, >> > > and >> > > > > really low bandwidth usage. If we sent 1000 mails a day, it's a >> > lot. >> > > > > >> > > > > ...and yes, they even pay their bills, though not me very well :) >> > > > > >> > > > > I think it's which SSL algorithm is being used that's at least >> > > partially >> > > > > to blame. I have: >> > > > > SSL_Version: SSLv23:!SSLv3:!SSLv2 >> > > > > SSL_Cipher_list: kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4 >> > > > > :!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:! >> > > > > CAMELLIA128:!IDEA:!SEED >> > > > > >> > > > > I tried the default wih SSL_Cipher_List blank before, I don't >> think >> > > > there >> > > > > was a difference (but I've played with so many settings, I really >> > > don't >> > > > > remember) >> > > > > >> > > > > And last, on the SSL buffer size. If set to zero in the gui, on >> > > windows >> > > > > 2012, it says in green that it's set to 64 MB. I follow what >> you're >> > > > saying >> > > > > about it readying 4x 16 Kb without a loop cycle. Is that a good >> or >> > > bad >> > > > > thing though? >> > > > > >> > > > > >> > > > > >> > > > ------------------------------------------------------------ >> > > > ------------------ >> > > > _______________________________________________ >> > > > Assp-test mailing list >> > > > Assp-test@lists.sourceforge.net >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test >> > > > >> > > > >> > > > >> > > > >> > > > DISCLAIMER: >> > > > ******************************************************* >> > > > This email and any files transmitted with it may be confidential, >> > > legally >> > > > privileged and protected in law and are intended solely for the use >> of >> > > the >> > > > >> > > > individual to whom it is addressed. >> > > > This email was multiple times scanned for viruses. There should be >> no >> > > > known virus in this email! >> > > > ******************************************************* >> > > > >> > > > >> > > > ------------------------------------------------------------ >> > > > ------------------ >> > > > >> > > > _______________________________________________ >> > > > Assp-test mailing list >> > > > Assp-test@lists.sourceforge.net >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test >> > > > >> > > > >> > > ------------------------------------------------------------ >> > > ------------------ >> > > _______________________________________________ >> > > Assp-test mailing list >> > > Assp-test@lists.sourceforge.net >> > > https://lists.sourceforge.net/lists/listinfo/assp-test >> > > >> > > >> > > >> > > >> > > DISCLAIMER: >> > > ******************************************************* >> > > This email and any files transmitted with it may be confidential, >> > legally >> > > privileged and protected in law and are intended solely for the use of >> > the >> > > >> > > individual to whom it is addressed. >> > > This email was multiple times scanned for viruses. There should be no >> > > known virus in this email! >> > > ******************************************************* >> > > >> > > >> > > ------------------------------------------------------------ >> > > ------------------ >> > > >> > > _______________________________________________ >> > > Assp-test mailing list >> > > Assp-test@lists.sourceforge.net >> > > https://lists.sourceforge.net/lists/listinfo/assp-test >> > > >> > > >> > ------------------------------------------------------------ >> > ------------------ >> > _______________________________________________ >> > Assp-test mailing list >> > Assp-test@lists.sourceforge.net >> > https://lists.sourceforge.net/lists/listinfo/assp-test >> > >> > >> > >> > >> > DISCLAIMER: >> > ******************************************************* >> > This email and any files transmitted with it may be confidential, >> legally >> > privileged and protected in law and are intended solely for the use of >> the >> > >> > individual to whom it is addressed. >> > This email was multiple times scanned for viruses. There should be no >> > known virus in this email! >> > ******************************************************* >> > >> > >> > ------------------------------------------------------------ >> > ------------------ >> > >> > _______________________________________________ >> > Assp-test mailing list >> > Assp-test@lists.sourceforge.net >> > https://lists.sourceforge.net/lists/listinfo/assp-test >> > >> > >> ------------------------------------------------------------ >> ------------------ >> _______________________________________________ >> Assp-test mailing list >> Assp-test@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/assp-test >> >> >> >> >> DISCLAIMER: >> ******************************************************* >> This email and any files transmitted with it may be confidential, legally >> privileged and protected in law and are intended solely for the use of the >> >> individual to whom it is addressed. >> This email was multiple times scanned for viruses. There should be no >> known virus in this email! >> ******************************************************* >> >> >> ------------------------------------------------------------ >> ------------------ >> >> _______________________________________________ >> Assp-test mailing list >> Assp-test@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/assp-test >> >> >
------------------------------------------------------------------------------
_______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
