Re: [sniffer] Bad Rule - 828931
Dear Pete, In the future, please let us know immediately when you become aware of this. As it is, I will spend the next 3 hours picking out the fales positives from the mailbox and forwarding them to the clients. If I could have put the rulepanic in place an hour ago it would have saved me a lot of work and confused customers. Thank you, Michael Stein Computer House - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: sniffer@sortmonster.com Sent: Tuesday, February 07, 2006 4:07 PM Subject: [sniffer] Bad Rule - 828931 Hello Sniffer folks, I'm sorry to report that another bad rule got past us today. The rule has been removed (was in from about 1200-1500), but it may be in some of your rulebases. To avoid a problem with this rule you can enter a rule-panic entry in your .cfg file for rule id: 828931 If it is not already, the rule will be gone from your rulebase after your next update. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) Chief Scientist (www.armresearch.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bad Rule - 828931
On Tuesday, February 7, 2006, 6:15:13 PM, David wrote: DS Sorry, wrong thread on the last post. DS Add'l question. Pete, what is the content of the rule? The rule info is: Rule - 828931 NameC%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Created 2006-02-07 Source C%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Hidden false Blocked false Origin User Submission TypeManual Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength3.84258274153269 False Reports 0 From Users 0 Rule belongs to following groups [252] Problematic The rule was an attempt to build an abstract matching two ed pill names (you can see them in there) while compensating for heavy obfuscation. The mistake was in using %+ through the rule. The rule would match the intended spam (and there was a lot of it, so 22,055 most likely includes mostly spam. Unfortunately it would also match messages containing the listed capital letters in that order throughout the message. Essentially, if the text is long enough then it will probably match. A greater chance of FP match if the text of the message is in all caps. Also if there is a badly coded base64 segment and file attachment (badly coded base64 might not be decoded... raw base64 will contain many of these letters in mixed case and therefore increase the probability of matching them all). Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Bad Rule - 828931
So, in my terms (simple), this rule only catches msg if the two drug names are in that order and in all capitals, but not necessarily one immediately following the other? John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, February 07, 2006 6:44 PM To: David Sullivan Subject: Re: [sniffer] Bad Rule - 828931 On Tuesday, February 7, 2006, 6:15:13 PM, David wrote: DS Sorry, wrong thread on the last post. DS Add'l question. Pete, what is the content of the rule? The rule info is: Rule - 828931 NameC%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Created 2006-02-07 Source C%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Hidden false Blocked false Origin User Submission TypeManual Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength3.84258274153269 False Reports 0 From Users 0 Rule belongs to following groups [252] Problematic The rule was an attempt to build an abstract matching two ed pill names (you can see them in there) while compensating for heavy obfuscation. The mistake was in using %+ through the rule. The rule would match the intended spam (and there was a lot of it, so 22,055 most likely includes mostly spam. Unfortunately it would also match messages containing the listed capital letters in that order throughout the message. Essentially, if the text is long enough then it will probably match. A greater chance of FP match if the text of the message is in all caps. Also if there is a badly coded base64 segment and file attachment (badly coded base64 might not be decoded... raw base64 will contain many of these letters in mixed case and therefore increase the probability of matching them all). Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bad Rule - 828931
Pete, Gotcha. Basically anything that I trapped that is over 10 KB may have failed this (because that would be indicative of having an attachment in base64). It is much less likely to have hit on things without attachments, but it of course would be possible, and the bigger it was, the more likely that it could have failed. I also searched my Sniffer logs for the rule number and found no hits. It appears that I missed the bad rulebase. Thanks, Matt Pete McNeil wrote: On Tuesday, February 7, 2006, 6:15:13 PM, David wrote: DS Sorry, wrong thread on the last post. DS Add'l question. Pete, what is the content of the rule? The rule info is: Rule - 828931 NameC%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Created 2006-02-07 Source C%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Hidden false Blocked false Origin User Submission TypeManual Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength3.84258274153269 False Reports 0 From Users 0 Rule belongs to following groups [252] Problematic The rule was an attempt to build an abstract matching two ed pill names (you can see them in there) while compensating for heavy obfuscation. The mistake was in using %+ through the rule. The rule would match the intended spam (and there was a lot of it, so 22,055 most likely includes mostly spam. Unfortunately it would also match messages containing the listed capital letters in that order throughout the message. Essentially, if the text is long enough then it will probably match. A greater chance of FP match if the text of the message is in all caps. Also if there is a badly coded base64 segment and file attachment (badly coded base64 might not be decoded... raw base64 will contain many of these letters in mixed case and therefore increase the probability of matching them all). Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bad Rule - 828931
Pete, The overflow directory disappeared when 3.x was introduced. I posted a follow up on the Declude list about how to do this. Matt Pete McNeil wrote: On Tuesday, February 7, 2006, 8:14:53 PM, David wrote: DS Hello Pete, DS Tuesday, February 7, 2006, 8:11:50 PM, you wrote: DS Not sure, can anyone think of a way to cross check this? What if I put DS all the released messages back through sniffer? PM That would be good -- new rules were added to correctly capture the PM bad stuff. I almost suggested something more complex. DS That said...anyone know specifics of reprocessing messages through DS Declude on Imail? I know that in 1.x Declude would drop some kind of DS marker so that q/d's copied into spool would not be reprocessed but I DS don't remember what it was and don't know if it works same in 3.x. DS Posted question on Declude JM list but no answer so far. IIRC messages in the spool under scan would be locked until declude was done with them. After that, placing the Q and D files into the spool would mean that normal IMail processes would deliver them on the next sweep. The way around this was to place the messages back in the overflow folder (I'm not sure which parts - I think the Q goes in overflow and the D stays in spool -- someone will know for sure). The theory there is that messages sent to the overflow folder are sent there before they are scanned in order to backlog the extra processing load. So, messages coming out of the overflow folder would naturally be scanned ( for the first time - thinks the robot ). _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Bad Rule - 828931
Thanks for the update, Pete. I also appreciate that you expanded on how that rule went wild. I can see that the intent was good but the unintended consequences were not so good. Here's how it played out on my server: How many messages hit the FP rules: 2,042 How many messages Declude decided were ham anyway: 1,093 How many messages Declude decided were viruses: 0 How many messages Declude decided were spam: 949 Of the spam, when re-queued, how many were ham: 583 Of the spam, when re-queued, how many were still spam: 366 So, in total: How many messages hit the bad 828931 rule: 2,042 How many were indeed spam: 366 How many were false positives: 1,676 Andrew 8) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Bad Rule - 828931
Thanks for the update, Pete.I also appreciate that you expanded on how that rule went wild. I can see that the intent was good but the unintended consequences were not so good.Here's how it played out on my server:How many messages hit the FP rules: 2,042How many messages Declude decided were ham anyway: 1,093How many messages Declude decided were viruses: 0How many messages Declude decided were spam: 949Of the spam, when re-queued, how many were ham: 583Of the spam, when re-queued, how many were still spam: 366So, in total:How many messages hit the bad 828931 rule: 2,042How many were indeed spam: 366How many were false positives: 1,676Andrew 8)p.s. Re-posted in HTML so that I don't have to explain the line breaks that were eaten in the plain text version post.
