RE: [Declude.JunkMail] Tuning Declude
I agree with Scott that Dan has asked some very good questions, not only in this thread, but others as well. And unlike me when I first started with Declude, he is asking the *right* questions. Keep at it, Dan, since you are on the right track. Your questions are helping others as well (myself included). Dan Horne, CCNA Systems Administrator TAIS Web Wilcox World Travel Tours [EMAIL PROTECTED] CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of R. Scott Perry Sent: Thursday, February 13, 2003 6:44 PM To: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] Tuning Declude I don't pretend to fully understand Declude (others are far more wise than I). However, I use fairly standard settings with the exception of using SNIFFER (weight 17) for enhanced spam detection. What is SNIFFER? I can't find any mention of it in the Declude.JunkMail manual, http://www.declude.com/JunkMail/manual.htm. There is however a reference to it in both GLOBAL.CFG and $default$.junkmail. Is SNIFFER the same as Mesage Sniffer, http://www.sortmonster.com/? They are one and the same. The test name is SNIFFER, the product name is Message Sniffer. It is a third party program used to detect spam, that can be hooked into Declude JunkMail. SPAM-NONE weightrange x x 0 4 SPAM-VLOW weightrange x x 5 9 SPAM-LOW weightrange x x 10 14 SPAM-MID weightrange x x 15 19 SPAM-HIGH weightrange x x 20 29 SPAM-VHIGH weight x x 30 0 So if I am understanding what you have written above correctly it looks like you have used the flexibility of Declude.JunkMail to create your own tests. You created these tests by adding them to GLOBAL.CFG. Is that correct? Are there tests that you have created being referenced again in $default$.junkmail below? Should the last line read SPAM-VHIGH weightrange x x 30 0? Those are custom tests, which were added to the global.cfg file. Those lines define the tests (test definitions always go in the global.cfg file); to use them on incoming E-mail, you would need a corresponding line in the \IMail\Declude\$default$.JunkMail file (such as SPAM-HIGH HOLD, which would quarantine any E-mail failing the SPAM-HIGH test, which in this case is E-mail with a weight between 20 and 29). The last line is correct -- it defines a test that gets triggered with a weight of 30 or higher (with no upper limit). What is the significance of having a range 30 0 in the last line? Does that just mean 30+, an open-ended range? Actually, I wouldn't worry about that one right now. The weightrange test type is a bit different, as it has the low and high weights in the test definition. Normally, the last number in a test definition is 0, which is the weight that gets added to E-mail that does NOT fail the test (a fairly confusing topic, which is why it probably is best not to worry about it). What are the x's for in the test definitions? Those are used as placeholders. A test definition has the test name followed by the test type, then 2 pieces of test-specific information, followed by the weight for the test, and the 0. An x is used for the test-specific information for tests that don't need such information (such as the weight tests). SPAM-MID SUBJECT SPAM-MID SPAM-HIGH SUBJECT SPAM-HIGH SPAM-VHIGH SUBJECT SPAM-VHIGH In the above section, in the last 3 lines, what is the significance of having the test repeated at the end of the line, e.g. SPAM-MID SUBJECT SPAM-MID. With the SUBJECT action, you can specific the text to use to alter the subject. In this case, an E-mail failing the SPAM-MID test would have SPAM-MID added to the subject. If you are holding on any messages with a weight of 30+ shouldn't the last line read SPAM-VHIGH HOLD? Yes. :) I wish there was something like that too. I don't feel like I'm going to be able to figure out this program out in the time alotted for the trial period. If you need more time to fully evaluate Declude JunkMail, be sure to let me know, as we can extend the evaluation period in most cases if necessary. What is the different between an incoming action and an outgoing action? Incoming actions are used on incoming E-mail (what you would normally use, and what most people expect from a mailserver anti-spam program). An outgoing action is the action to take on outgoing E-mail that fails certain spam tests. Outgoing actions are rarely used (they might be used if you were worried that your users were going to send spam).
RE: [Declude.JunkMail] how much is junk?
The average spam/ham ratio for reported logs in Message Sniffer is 70%-75%. That is, 70%-75% of messages on average are spam. This is a small sample (about 20 systems on average) but it has been a very consistent range. _M | -Original Message- | From: [EMAIL PROTECTED] | [mailto:[EMAIL PROTECTED]] On Behalf Of paul | Sent: Thursday, February 13, 2003 2:37 PM | To: [EMAIL PROTECTED] | Subject: [Declude.JunkMail] how much is junk? | | | Ok guys, what do you see in ratio of junk vs good mail per | day? Do you get | more junk than legit? Here I notice we're killing more than | 50% of incoming | mail. Average messages processed per day range from 13K to | 23K. Using the | log analyzer I found that January we processed 615,082 | messages, and 53% we | deleted by Declude, that's alot! | | Granted, near daily updates to my kill file and filters help boost the | number. | | Paul | | | --- | [This E-mail scanned for viruses by Declude Virus] | | --- | [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] how much is junk?
We've here a constant value of 10 - 12% during workday and around 30% on weekends. This because during weekends there are not so much real messages. The values are based on a system with 350 domains 650 mailboxes and 2500 incomming messages/day. Reporting from 01/01/2003 to yesterday on my personal mailbox: I recieve around 54 messages per day (without control, notification and list-messages) 1.5 of them whas spam that passed the declude-tests 12.5 of them was hold by declude. Based on this values and other 7 monitored mailboxes on our system I can assume that we hold 88 to 90% of incomming spam. What can be the cause that we have only such a low spam ratio? 2/3 of our domains are from italy (.it tld) - International spammers filter out country tld's? On the other side we've a lot of italian spam and nearly 0 german spam, even if over 90% of our clients are german. - country spammers filter for it-domains! The average lifetime of our domains and their mailboxes is 1.5 year. This should be time enough to be found from spammers. Markus -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Scott MacLean Sent: Thursday, February 13, 2003 11:17 PM To: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] how much is junk? I have a user that receives over 10,000 spam emails a day. I personally get about 500 spam to 50 real emails a day. Of those, typically around 5-10 get past Declude. At 02:46 PM 2/13/2003, Helpdesk wrote: on 2/13/03 2:36 PM, paul wrote: Ok guys, what do you see in ratio of junk vs good mail per day? Spam messages account for over 75% of our incoming messages (we're an ISP). Later, Greg --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. ___ Scott MacLean [EMAIL PROTECTED] ICQ: 9184011 http://www.nerosoft.com --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Tuning Declude
Hey, Scott, The first thing to remember is that no one spam test is perfect, and all spam tests will have some false positives (which is what the weighting system really helps out with). Yes, that concept is pretty easy for me to understand from an intellectual level. I mean if there was a perfect spam test, then you only need one test. And everyone would use it. And there would be no more spam, nor people making money on selling spamming products. :-) False Positives === BADHEADERS II In this case, you also need to determine exactly what a false positive is. Is it a false positive if the sender uses a broken mail client, and IMail rejects the message, for example? Yes, I would consider that a false positive. For me, I'm not even really going to trouble myself with why the BADHEADERS test is creating a false positive. Even if the headers are bad and can be fixed by the sending party I can't see any way where I'm going to track individual vendors and have them fix their software. Heck, I know that some of the BADHEADERS I see are generated by a 3rd-party server monitoring utility that we use and I have in the past spoken with developer directly and I can definitely say that he will NOT fix his program so that the e-mail it generates pass the BADHEADERS test. From my point of view, a false positive is a false positive is a false positive. I just need to make sure that a message has to fail more than the BADHEADERS test to get rejected. The only legitimate mail that the BADHEADERS test catches is mail that has broken headers (which may never reach you anyways). Whenever legitimate E-mail fails the BADHEADERS test, I strongly recommend fixing the problem. In most cases, blocking based on the BADHEADERS test alone is very useful. So how would you recommend fixing the problem? NOABUSE NOPOSTMASTER II Note that these catch all E-mail from @yahoo.com, @hotmail.com, and some other major domains, which accounts for their high false positives. And the reason that they catch that stuff is because Yahoo.com and Hotmail.com don't support the e-mail addresses abuse@domain.com and postmaster@domain.com? REVDNS I SPAMHEADERS ... and these will always have high false positives, but are very good at detecting systems that are likely to send out spam (IE it may be sending legitimate mail today, but is likely to send spam tomorrow). Is that because they would be used as mail relays? Quandry #1) How to use Declude.JunkMail to weight messages from a technical standpoint I understand the concept of weighting the e-mails from an abstract level but it's not clear to me from a technical level how Declude implements it. This one is fairly straightforward. Each test has a weight assigned to it. Declude JunkMail runs an E-mail through all the spam tests, and adds the weights of each test that the E-mail fails. What does it mean when there is no weight at the end of the X-SPAM-TESTS-FAILED line? There are big holes in my understanding of the purpose of the global.cfg vs. the $default$.junkmail files. Is there a step-by-step breakdown of each line of global.cfg somewhere that I can read? I've been reading the JunkMail Manual and it makes mention of different entries as needed but there doesn't seem to be a comprehensive explanation of the cfg as a whole. From the manual: --- \IMail\Declude\global.cfg - This contains the global Declude JunkMail settings. It includes standard settings (such as whether to add the spool file name to the E-mail headers with the XSPOOLNAME option), test definitions (which you shouldn't normally need to change or worry about), and the actions to take for outgoing E-mails (for the Pro version). This is not a script (Declude doesn't use scripts), it is a standard configuration file, so there is no order to it (if adding lines to it, you can put them anywhere in the file). I had read this paragraph (and the corresponding paragraph for $default$.junkmail) and to be honest a general paragraph about the purpose of the file wasn't as helpful to me as a line by line breakdown of each entry in the file would've been. I had started at the top of GLOBAL.CFG and would read down one line at a time. If got to something I didn't understand I would do a CTRL-F and try to search for it on the Declude.JunkMail Manual, http://www.declude.com/JunkMail/manual.htm, and I didn't get very far before I got to LOGLEVEL, the 13th line down. When I searched for it I found 2 references for it in description's of how to do other things but nothing else. What I was expecting to find was an interation of all the different possible settings for LOGLEVEL, e.g. LOW MID HIGH (I still don't even know for sure if that's all of them!), and what specifically is added and/or substracted from the log when you set LOGLEVEL to each one of those settings. Perhaps that document does exist and I just
Re: [Declude.JunkMail] Tuning Declude
Howdy, Scott, SPAM-NONE weightrange x x 0 4 SPAM-VLOW weightrange x x 5 9 SPAM-LOW weightrange x x 10 14 SPAM-MID weightrange x x 15 19 SPAM-HIGH weightrange x x 20 29 SPAM-VHIGH weight x x 30 0 So if I am understanding what you have written above correctly it looks like you have used the flexibility of Declude.JunkMail to create your own tests. You created these tests by adding them to GLOBAL.CFG. Is that correct? Are there tests that you have created being referenced again in $default$.junkmail below? Should the last line read SPAM-VHIGH weightrange x x 30 0? Those are custom tests, which were added to the global.cfg file. Those lines define the tests (test definitions always go in the global.cfg file); to use them on incoming E-mail, you would need a corresponding line in the \IMail\Declude\$default$.JunkMail file (such as SPAM-HIGH HOLD, which would quarantine any E-mail failing the SPAM-HIGH test, which in this case is E-mail with a weight between 20 and 29). The last line is correct -- it defines a test that gets triggered with a weight of 30 or higher (with no upper limit). What is the significance of having a range 30 0 in the last line? Does that just mean 30+, an open-ended range? Actually, I wouldn't worry about that one right now. The weightrange test type is a bit different, as it has the low and high weights in the test definition. Normally, the last number in a test definition is 0, which is the weight that gets added to E-mail that does NOT fail the test (a fairly confusing topic, which is why it probably is best not to worry about it). Believe it or not, I actually think I do understand what's going on with the weighting definitions in the tests. Like these 2 default tests from GLOBAL.CFG... MAILFROMenvfrom x x 12 0 IPNOTINMX ipnotinmx x x 0 -3 The first field is the name of the test. The second field is the type of te st. The third and fourth fields are placeholders so all test definitions can have the same number of fields, i.e. 6 fields, the maxiumum length needed for an ip4r type test. The fifth field is the weight that is added to the total weight if a particular messages fails a test and the sixth field is the weight that is added to the total weight if a message does NOT fail the test. From what I can tell only IPNOTINMX is the only test which benefits from the sixth field, but that's alright. I'm sure you had your reasons for adding it. I'm sure future situations would arise where that might come in handy for other existing (or yet to be defined) tests. Or is an incoming action applied when a message is going from a user on the Internet to a user on the IMail server and an outgoing action is applied when a message is going from a user on the IMail server to a user on the Internet? That is correct. If the latter what sort of actions come into play if a message is being sent from an IMail user to an IMail user? In this case, it will be treated as an incoming E-mail. Forgive me if this is all basic stuff but I'm just trying to fill in the gaps. These are some very good questions, and I'm hoping other people new to Declude JunkMail read this, to help them learn about Declude JunkMail. It also helps point out areas where the manual may need improvement. -Scott Glad to see I understand something! Thanks for the encouragement to answer questions. It is very helpful to know that all questions, regardless of level, are welcome here. Dan This E-mail is scanned and free from viruses. www.nexustechgroup.com --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Tuning Declude
Dan, Sniffer has made a huge difference for us. We weight the test a 12 and flag emails as Spam at 15. We only ran for a couple of months without it, but I watch our logs very closely and the benefit of using Sniffer is significant. Sniffer is an entirely different type of test from Declude. It tests the content of the email for identifiable strings, phone numbers, URLs, email addresses, etc that will only be found in emails from known spammers. Most people on this list including myself highly recommend adding the Sniffer product. The Declude/Sniffer combo is a match made in heaven. Bill -Original Message- From: Dan Geiser Sent: Fri, 14 Feb 2003 14:45:06 -0500 Subject: Re: [Declude.JunkMail] Tuning Declude Hello, All, For most of you who use Message Sniffer: Do you find that using it along with the default testsWEIGHT10 and WEIGHT20 are sufficient for your needs? How integral of an addition to Declude.JunkMail is Message Sniffer? Does it make an earth-shattering difference in what your spam-filtering, does it just add an additional level of nuance that can't be gotten through the tests which Declude has, or is it just an entirely different type of test? What made you decide to add Message Sniffer into the mix for your Declude installation? How long did you run Declude.JunkMail without SNIFFER before putting it into play? Thanks For Your Time, Dan - Original Message - From: Bill Newberg [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, February 13, 2003 7:19 PM Subject: RE: [Declude.JunkMail] Tuning Declude What is SNIFFER? I can't find any mention of it in the Declude.JunkMail manual, http://www.declude.com/JunkMail/manual.htm. There is however a reference to it in both GLOBAL.CFG and $default$.junkmail. Is SNIFFER the same as Mesage Sniffer, http://www.sortmonster.com/? They are one and the same. The test name is SNIFFER, the product name is Message Sniffer. It is a third party program used to detect spam, that can be hooked into Declude JunkMail. I added Sniffer to Declude JunkMail recently and I am very pleased. It is a great addition to Declude. Regards, Bill Newberg This E-mail is scanned and free from viruses. www.nexustechgroup.com --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Tuning Declude
Believe it or not, I actually think I do understand what's going on with the weighting definitions in the tests. Like these 2 default tests from GLOBAL.CFG... MAILFROMenvfrom x x 12 0 IPNOTINMX ipnotinmx x x 0 -3 The first field is the name of the test. The second field is the type of te st. The third and fourth fields are placeholders so all test definitions can have the same number of fields, i.e. 6 fields, the maxiumum length needed for an ip4r type test. The fifth field is the weight that is added to the total weight if a particular messages fails a test and the sixth field is the weight that is added to the total weight if a message does NOT fail the test. Perfect. :) From what I can tell only IPNOTINMX is the only test which benefits from the sixth field, but that's alright. I'm sure you had your reasons for adding it. I'm sure future situations would arise where that might come in handy for other existing (or yet to be defined) tests. For a long time, no test used the sixth field -- it was originally added because it *sounded* like a good idea. Only recently was the IPNOTINMX test added, that benefited from the sixth field. -Scott --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
