Re: [Declude.JunkMail] RE: On RFC Violation - Declude allows attachments and Virus to pass through untouched and unscanned

[Matt]

Andy, Declude introduced NONSTANDARDHDR as a vulnerability in 4.2.20.  This was their answer to handling the "broken spam" issue, but it is not an appropriate fix since this throws the message into your Virus directory which is typically outside of one's review process.  There was an assumption by Declude that this was all spam at the time that they introduced this as a fix for the broken header thing (after they failed to find a solution for resolving the header issues). Here's what Scott Fisher said in response to my claim that this wasn't all spam: "I certainly regularly receive incorrectly formatted email. I'm pretty small volumne, but looking over my logs (I have an external test for this condition), it is 111 non-spam messages this month. My email volume is pretty low. But I'm not looking forward to hand correcting 120 of these a month. " I don't have stats on this, I just know that I don't need to be blocking this.  In fact, the broken spam doesn't even get to Declude but very rarely since greylisting takes care of it.  I would have just about a 100% FP rate on this filter. Now according to Michael's tests, the CR-only pattern leads to parsing issues in Declude Virus where it can't even find the attachment to scan it. I think the best approach is to read the E-mail into memory, convert the main copy in memory to CRLF if a non-CRLF pattern is found anywhere and store another original copy, set a value to trigger a Declude JunkMail test if the pattern is present in the headers, and if that is set, when it comes time to write the message, use the CRLF patterned headers and the original body that way you aren't rewriting someone's purposefully formated LF-only or CR-only pattern. Unfortunately all it takes to trigger this is a Perl or PHP programmer doing a message form and using \n being not aware of the RFC's, but being consistent with Linux, and instead of the \r\n pattern that it should be. I can't block E-mail based simply on this one pattern because it will capture good E-mail regularly, and do so outside of my spam review system.  Also, there are dozens of RFC violations that are not only common, but also tolerated by mail servers and E-mail clients, and outside of Declude, this is one of those things that is mostly tolerated.  The reasoning to treat it as a vulnerability was not because it was, but because they couldn't figure out a way to rewrite the headers properly.  I'm confused as to why unless they just simply didn't want to touch the code involved.  I do hope that they go back and figure out how to make that work otherwise I will be faced with blocking legitimate E-mail in a non-reviewable area, or potentially passing viruses completely unscanned.  That's not a good set of choices. Matt Andy Schmidt wrote: Hi Matt, I'm not sure that the issue is attachments. There is nothing wrong with attachments using Unix/Apple linefeeds. But the RFCs for SMTP (and similar protocols) all require that each header line must end with CRLF and each header line ends with CRLFCRLF. Anything else is not a legitimate SMTP mailer - but rather a poor attempt in faking one. We agree that in the headers CR only or LF only should be "treated" by Declude as if they were CRLF (and detected as header violations, nevertheless). I also have no problem if Declude were to "fix" those linefees to proper CRLF if it so desires. Best Regards Andy Schmidt Phone: +1 201 934-3414 x20 (Business) Fax: +1 201 934-9206 -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Monday, October 23, 2006 12:27 AM To: declude.junkmail@declude.com Subject: Re: [Declude.JunkMail] RE: On RFC Violation - Declude allows attachments and Virus to pass through untouched and unscanned Unfortunately, treating everything with only CR's or only LF's as a virus/vulnerability is not perfectly accurate. I definitely see legitimate E-mail coming this way. I believe the best solution is to just simply treat either CR, LF, or CRLF as line breaks when parsing the message in Declude for virus scanning, header modification, etc. Linux uses LF-only by default for line breaks in text files, and this is why this is common in both spamware and in homegrown E-mail scripts (which may well be legitimate). In fact, if someone attaches a text file created in Linux to an E-mail message, it should come through with a LF-only pattern in the text attachment, and that certainly isn't a virus. CR-only is used by at least old Macs (OSX is now essentially BSD so it's LF-only now), and I haven't seen anyone use this format for constructing E-mail, and I have my doubts about whether it is supported by E-mail clients if it was used in the MIME headers, but of course attachments could come as CR-only and be fully legitimate, though not necessarily displayable as anything but a single line in non-old Mac E-mail clients. Detecting and correcting for this definitely seems easy to do. In _vbscript_, this is how I approach it with Linux format (LF-only): ' Correct for non-CRLF linebreaks. regEx.Global = True regEx.IgnoreCase = False regEx.Pattern = "[^\r]\n" ' If regEx.Test(strMessage) Then boolTestBadCrLf = True strMessage = Replace(strMessage, vbLf, vbCrLf) strMessage = Replace(strMessage, vbCr & vbCrLf, vbCrLf) End If There are fancier ways to do the replacement as well, but this seemed appropriate for _vbscript_, and it only kicks in when the pattern is found, and that is rare. This also only affects what is in memory and not the file. It has been stated in the past that Declude tried resolving this much earlier when it was pointed out that the headers were being written to the end of the body. I don't see why this would be the case if a simple check and replace was done immediately after reading in the file's contents, and before all tests and before writing the headers. I do wish Declude would respond to this because it does concern me now that there is a parsing error that leads to huge vulnerability in Declude, and while there is currently a method of blocking such messages in Declude with a vulnerability switch, I know that this is not a universally accurate method, and I fear that it could tag things such as Linux style text attachments. Matt Andy Schmidt wrote: Hi, Well, the necessary logic seems absolutely simple: A) parse token, up to EITHER CR or LF B) if "CR" found, see if followed by LF, skip over CF/LF and start new line C) if only "CR" found, and not followed by LF, skip over CR and start new line. Set SMTP Header violation flag. D) if "LF" found, skip over LF and start new line. Set SMTP Header violation flag. The key is to detect an "apparent" end of headers - no matter if it's a proper CRLFCRLF sequence, or if it's only a CRCR or LFLF or CRLFLF combination so that spam can be recognized as such, virus can't be hidden in none-standard line feeds and Declude doesn't attach headers behind the body of the message. I'm certain that we are looking at one cause why obvious spam is currently slipping past Declude in the current releases. Best Regards Andy Schmidt Phone: +1 201 934-3414 x20 (Business) Fax: +1 201 934-9206 -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Friday, October 20, 2006 05:35 PM To: declude.junkmail@declude.com Subject: Re: SPAM-WARN: Re: [Possible Spam][Declude.JunkMail] On RFC Violation - Declude allows attachments and Virus to pass through untouched and unscanned I think that the real issue is that if Declude just simply understood CR patterns, the messages could be properly parsed, and headers could be properly inserted. There is a similar and longstanding issue with Declude failing to decode base64 encoding in very long lines (i.e. +1000 or so characters long). This also leads to failing to scan the attachments. One virus has been spreading for the last year and a half with this symptom, and the only reason why my Declude virus stops it is because it always sends as an EXE, but this causes backscatter. If this was changed to do something like a ZIP file, which most of us don't block, it could result in viruses passing straight through Declude. While Declude does have a new option to treat CR patterns as a vulnerability, this isn't something that I can use since there is unfortunately a steady, yet small, stream of legitimate messages that have this flaw. The flaw itself doesn't represent a vulnerability, in fact this wouldn't be a vulnerability at all if Declude understood how to parse the messages. I can't see how fixing this flaw, or the base64 long line flaw would take them any more than a couple of hours. I have a plug-in that I wrote for Declude that is fully capable of understanding CR patterns as well as long base64 code without issue. Matt Michael Thomas - Mathbox wrote: David, In my opinion, which others may not share, Declude should detect all RFC/MIME violations and flag them in some manner. There exist quite a few that are common to spam messages, but not flagged by Declude. However, that is a totally different subject than the point of my test. The RFC violation was simply a symptom. Some admins might not choose to delete or block messages with that construction. There exist well-known web sites that generate response email messages where one or two lines out 50-100 are missing one of the Cr/Lf pair characters. The point is that for many (maybe most) people on this list, Declude is the lock and keys for securing email passing through their mail servers. Declude trusts that the email message will be well-formed. Because of that mis-placed trust, Declude did not reliably detect any attachment, regardless of type, and therefore did not invoke the scanners. Note that this test was performed on Declude version 3.1.1. Maybe, the new gateway product is not quite so trusting. I have no idea. We are an ISP and would not pay the fees associated with ISP use of the gateway product. In any event, this is a well-known issue and has been for some time. I reported this issue to Declude and the list some time ago regarding spam not being scanned because of this issue. At that time, I was so focused on the spam problem, I did not think about the attachment/virus side of the issue, which should have been obvious to me. Michael Thomas Mathbox 978-683-6718 1-877-MATHBOX (Toll Free) -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of David Sullivan Sent: Friday, October 20, 2006 3:51 PM To: declude.junkmail@declude.com Subject: SPAM-WARN: Re: [Possible Spam][Declude.JunkMail] On RFC Violation - Declude allows attachments and Virus to pass through untouched and unscanned Hello Michael, Thanks for the great research. Wouldn't this be the purpose of Vulnerability detection in Declude? "Declude detects mal-formed messages that can allow viruses to be hidden from email server virus scanners." We treat all vulnerabilities as viruses, send the notice and 86 the message. -David Thursday, October 19, 2006, 10:52:25 PM, you wrote: MTM> Hi All, MTM> Well, when responding on declude.junkmail@declude.com to Will about RFC MTM> violations, I said I would test this and I did. MTM> -------------------- MTM> While writing this message, I happened to think about attachments. It would MTM> appear to me, that there is an implied possibility for attachments and MTM> therefore viruses to pass through undetected. All that should be required is MTM> that the lines that make up the entire email, including the attachment MTM> section, be terminated with line feeds instead of carriage return/line feed MTM> pairs. Under such condition, Declude would see only one line and not find MTM> the relevant sections. I will test this possibility. MTM> -------------------- MTM> Tested: Declude v3.1.1 for IMail MTM> As it happens, my suspicions were accurate. I wrote a script that could be MTM> modified to remove either the carriage-returns or the line-feeds from a MTM> message file. I then created a message in Outlook Express, added an MTM> executable file (uptime.exe) as an attachment and saved it in my Draft MTM> folder. I then dragged that message to the same location as the script and MTM> renamed it to match the file name in the script (Rfc.eml) I ran the script, MTM> which stripped the carriage-returns and produced Rfc2.eml. I renamed MTM> Rfc2.eml to RfcNoCr.eml. In the script, I then changed vbCr to vbLf and ran MTM> it again, which stripped the line-feeds and produced Rfc2.eml. I renamed MTM> Rfc2.eml to RfcNoLf.eml. MTM> Now, to get IIS SMTP to actually process the file, you must edit each file MTM> and remove the single Cr or Lf and press the Enter Key, producing a CrLf MTM> pair after the To field and the From field. I also added the string "No Cr" MTM> to the end of the subject of RfcNoCr.eml and added No Lf to the subject of MTM> RfcNoLf.eml. So for example change: MTM> -------------------- MTM> From: "Michael Thomas - Mathbox" <[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]> [Cr]To: MTM> "[EMAIL PROTECTED]" <mailto:[EMAIL PROTECTED]> [Cr]Subject: Test Attachment Pass-Through on RFC MTM> Violation[Cr]<line continues> MTM> -------------------- MTM> Change To MTM> -------------------- MTM> From: "Michael Thomas - Mathbox" <[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]> MTM> To: "[EMAIL PROTECTED]" <mailto:[EMAIL PROTECTED]> MTM> Subject: Test Attachment Pass-Through on RFC Violation No Cr[Cr]<line continues>> MTM> -------------------- MTM> Now it so happens, a long time ago, I wrote a couple of tests to detect MTM> these RFC violations, so first I had to disable them in my GLOBAL.CFG, which MTM> I did by commenting them out. Note that I also BAN the .EXE extension and I MTM> left that enabled. MTM> Now copy and paste the two files into the pickup directory of your favorite MTM> IIS SMTP pickup directory. Viola, you just passed an executable through MTM> Declude and through your mail server. That executable could very well have MTM> been a virus. MTM> Note that Declude detected RfcNoLf.eml as [Outlook 'CR' Vulnerability]. Ok MTM> good. MTM> But Declude let RfcNoCr.eml pass straight through without calling the virus MTM> scanners, because Declude did NOT see an attachment. Also, because Declude MTM> did not see an attachment, Declude did not ban the .EXE extension. MTM> Here are the log entries from RfcNoLf.eml MTM> -------------------- MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Scanning Time: 218ms MTM> [kernel=31 user=187] MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Virus scanner 1 reports exit MTM> code of 0 MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Virus detected. Not continuing MTM> with remaining scanners. MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd 0: MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Starting EXT check . MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd MTM> C:\IMAIL\spool\proc\work\D1b2101b7000083ba.vir\*.* MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd 0 MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Deleted MTM> C:\IMAIL\spool\proc\work\D1b2101b7000083ba.vir\0. MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd report.txt MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Deleted MTM> C:\IMAIL\spool\proc\work\D1b2101b7000083ba.vir\report.txt. MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd han=13e9c0 b=False MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd File(s) are INFECTED [[Outlook MTM> 'CR' Vulnerability]: 0] MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd High code=23. MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd AV returned 23 MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Scanned: CONTAINS A VIRUS MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd From: [EMAIL PROTECTED] To: MTM> [EMAIL PROTECTED] [incoming from XX.XXX.XXX.X] MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Subject: Test Attachment MTM> Pass-Through on RFC Violation No Lf MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Skipping non-AV E-mail MTM> BANnotify.eml MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd MTM> C:\IMAIL\Declude\postmaster.eml MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Starting E-mail file MTM> C:\IMAIL\Declude\postmaster.eml MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd C:\IMAIL\IMail1.exe -h MTM> "mathbox.com" -t "[EMAIL PROTECTED]" <mailto:[EMAIL PROTECTED]> -u "[EMAIL PROTECTED]" <mailto:[EMAIL PROTECTED]> -s MTM> "Mathbox Email Virus Scanning detected and quarantined a virus" -f MTM> "C:\IMAIL\spool\proc\work\D1b2101b7000083ba.sm0" MTM> 10/19/2006 20:41:23.487 q1b2101b7000083ba.smd TempName = MTM> C:\IMAIL\Declude\postmaster.eml MTM> -------------------- MTM> Here are the log entries from RfcNoCr.eml MTM> -------------------- MTM> 10/19/2006 20:41:10.690 q1b2101da000083bb.smd Setting Scan File 1 to MTM> C:\Progra~1\FSI\F-Prot\FPcmd.exe /TYPE /SILENT /SERVER /NOMEM /ARCHIVE MTM> /NOBOOT /DUMB /REPORT=report.txt. MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd CFG: Setting report parse 1 to MTM> Infection. MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting Scan File 2 to MTM> C:\imail\declude\runclamscan.exe log=3 MTM> C:\clamav-devel\bin\clamscan.exe MTM> --quiet --no-summary --tempdir=c:\tmp\ MTM> --database=C:\clamav-devel\share\clamav\ --max-ratio=0 --mbox -l report.txt. MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd CFG: Setting report parse 2 to MTM> FOUND. MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting virus directory to: MTM> C:\IMAIL\spool\virus MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Incoming E-mail scanning MTM> turned ON MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Outgoing E-mail scanning MTM> turned ON MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting AVAFTERJM to ON. MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting MAXATONCE to 20. MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting scanner timeout to 120 MTM> seconds MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting AUTOFORGE to OFF. MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Scanner 0 Virus Codes: 3 6 8 9 MTM> 10 . OK Codes: MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Scanner 1 Virus Codes: 1 . OK MTM> Codes: MTM> 10/19/2006 20:41:10.908 q1b2101da000083bb.smd Skip Extensions: GIF TXT MPG MTM> PNG MTM> 10/19/2006 20:41:10.955 q1b2101da000083bb.smd 48 Ban Extensions: ADE ADP ASD MTM> ASP BAS BAT BIN CAB CHM CMD COM CPL CRT DLL EXE HLP HTA HTO INF INS ISP JS MTM> JSC JSE KSH LNK MDB MDE MSI OCX PCD PIF REG SCF SCR SCT SHB SHS SYS VB VBE MTM> VBS VBX VSMACROS VXD WSC WSF WSH MTM> 10/19/2006 20:41:11.002 q1b2101da000083bb.smd Virus Pro Registered MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd Starting locality check MTM> (sender=mathbox.com; nr=1 ca=off). nHas=1. MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd [EMAIL PROTECTED] [0-0] is MTM> local domain1 viaFM MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd Ending locality check MTM> (cached), sender=local. MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd Local host = mathbox.com MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd MTM> [EMAIL PROTECTED] Offset=5 MTM> Flags=1 MTM> 10/19/2006 20:41:11.033 q1b2101da000083bb.smd Msgid: MTM> 10/19/2006 20:41:11.049 q1b2101da000083bb.smd Subject: Test Attachment MTM> Pass-Through on RFC Violation No Cr MTM> -------------------- MTM> Here is the script to strip Cr or Lf, just change the vbCr below to vbLf. MTM> Just save it as: MTM> Rfc.vbs MTM> -------------------- MTM> Dim InFile MTM> Dim OutFile MTM> Dim Fso, File MTM> Dim AllText MTM> InFile = "Rfc.eml" MTM> OutFile = "Rfc2.eml" MTM> Set Fso = CreateObject("Scripting.FileSystemObject") MTM> If Fso.FileExists( InFile ) = True Then MTM> Set File = Fso.OpenTextFile( InFile, 1, False, 0 ) MTM> AllText = File.ReadAll MTM> File.Close MTM> Set File = Nothing MTM> AllText = Replace( AllText, vbCr, "" ) MTM> Set File = Fso.OpenTextFile( OutFile, 2, True, 0 ) MTM> File.Write AllText MTM> File.Close MTM> Set File = Nothing MTM> End If MTM> Set Fso = Nothing MTM> -------------------- MTM> Finally, if you want to test for these RFC violations, see MTM> http://www.mathbox.com/NoCrTest/NoCrTest.zip MTM> Michael Thomas MTM> Mathbox MTM> 978-683-6718 MTM> 1-877-MATHBOX (Toll Free) MTM> --- MTM> This E-mail came from the Declude.JunkMail mailing list. To MTM> unsubscribe, just send an E-mail to [EMAIL PROTECTED], and MTM> type "unsubscribe Declude.JunkMail". The archives can be found MTM> at http://www.mail-archive.com. -- Best regards, David mailto:[EMAIL PROTECTED] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.