RE: [Declude.JunkMail] Test needed along with sniffer
I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the spamtrap and hope that sniffer and surbl will do this. Do I even need surbl? Do you have so much workload on your mailserver that you need to downsize your spam-filter to one or two tests? Maybe http://www2.spamchk.com/public.htm will give you some answer. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Test needed along with sniffer
I am getting service timeouts due mostly to all the declude instances of traffic volume I handle about 2 messages a day, most of them during business hours I find that I accumulate declude processes that have consumed up to a minute of cpu time only to be idle and just sit there This also causes accumulated memory to be consumed I have been rebooting this server about twice a week I have also been spending time everyday adding to my filter files The server is a dual Xeon 2.4Ghz, 533 frontside bus with an Intel SATA raid card running Raid 10 It has about 100 small web site that do not get much traffic My goal is to reduce management time of the machine and to stabilize it so the need to reboot it is lessened I am prepared to put in a dual Xeon 3.4GH, etc but also want to make sure that I do not overkill Harry Vanderzand inTown Internet Computer Services 11 Belmont Ave. W. Kitchener, ON N2M 1L2 519-741-1222 Did you know we offer: - Province wide dial-up and high speed internet access - Web accessible email with anti-spam\antivirus protection - Computer hardware sales and service - Experienced website developers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Markus Gufler Sent: Wednesday, September 08, 2004 11:22 AM To: [EMAIL PROTECTED] Subject: RE: [Declude.JunkMail] Test needed along with sniffer I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the spamtrap and hope that sniffer and surbl will do this. Do I even need surbl? Do you have so much workload on your mailserver that you need to downsize your spam-filter to one or two tests? Maybe http://www2.spamchk.com/public.htm will give you some answer. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Test needed along with sniffer
Sniffer is very good. It detected 47600 out of 49250 spam messages for me through Sept 1-5. The SURBL filter contains a lot of body filters and can be CPU intensive. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 09/08/04 10:13AM I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the spamtrap and hope that sniffer and surbl will do this. Do I even need surbl? Any advice in this matter would be greatly appreciated. Thanks in advance Harry Vanderzand inTown Internet Computer Services --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Test needed along with sniffer
On Wednesday, September 8, 2004, 11:13:18 AM, Harry wrote: HV I am testing sniffer right now and wonder if I need to run all the other HV tests along side it. Well, you can probably get by without the other tests, but since you have Declude it would be MUCH better if you keep the other tests in place. Declude's strength is that it allows you to aggregate a variety of tests for greater accuracy. Sniffer is very, very good, but you will certainly see some benefit by using it along with other tests. HV I am trying to reduce my daily workload of analyzing the spamtrap and hope HV that sniffer and surbl will do this. Sniffer is perfect for that - particularly if you share your spamtrap data with us. Put another way, if you allow us to use your spamtrap then we will be taking over this work for you. All we need is POP3 account information and some details on how your spamtrap was formed so that we can properly classify it in our SPHUD (Spam Processing Heads Up Display). HV Do I even need surbl? Probably not. One of the AI elements in our robots crossreferences incoming spamtrap data with SURBL and other tests. More often than not we have the domain tagged before we see it in SURBL, and if we don't we grab it quickly. HV Any advice in this matter would be greatly appreciated. I recommend reviewing the Spam Test Quality Analysis: http://www2.spamchk.com/public.html You can use this to help tune your Declude configuration. I recommend applying the forumula: W = (a^2)100 Where (W) is the individual test weight (magnitude) based on test accuracy and (a) is the accuracy measured in the analysis (SA = spam-test accuracy, HA = ham-test accuracy). [ Regarding (magnitude), ham tests generate negative weights and spam tests generate positive weights. W will always be a positive value, so if you use an HA value for (a) then you will want to apply a negative W as your weight in Declude. ] For example, SNIFFER SA = 0.95, so W = ((0.95)^2)*100 = 90.25, Weight = 90. FIVETEN-SRC SA = 0.59, so W = ((0.59)^2)*100 = 34.81, Weight = 35. NOLEGITCONTENT HA=0.38, so W = ((0.38)^2)*100 = 14.44, Weight = -14 -- This test is measured when the test does not fail, so -14 must go in second weight column, not the first. If you use this analysis you should have your hold weight at or about 100. If you set your hold weight lower than 100, you will capture more spam at the risk of more false positives. If you set your hold weight higher than 100 you will have fewer false positives and more spam. !! This is research in progress - these formulas appear to work very well in preliminary testing. If you are already happy with your weighting system then you should probably stick with that until this theory has been tested further. !! We are developing a utility to do this work automatically. In the mean time, you can go through your test weights manually. You shouldn't have to do this frequently. Hope this helps, _M --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Test needed along with sniffer
Harry, Sniffer is a great addition to any Declude setup, however your issues are not due to just simply the size of your processors. We run a dual 1 GHz PIII system with RAID 5 and 5x10K Cheetahs, and we've managed to exceed 90,000 messages a day with dual virus scanners, and we could handle a bit more still. My thought is that you are either running a ton of BODY filters, a very slow virus scanner/scanners, or you are experiencing some form of I/O limitation. The idle processes also suggest that maybe there is an issue and an upgrade to a more recent version of Declude such as 1.79 or an interim release thereafter would be a good idea and most around here run them. You should be able to minimally do 10 times your current volume, so keep looking and keep describing your environment and a solution will likely come along. Matt Harry Vanderzand wrote: I am getting service timeouts due mostly to all the declude instances of traffic volume I handle about 2 messages a day, most of them during business hours I find that I accumulate declude processes that have consumed up to a minute of cpu time only to be idle and just sit there This also causes accumulated memory to be consumed I have been rebooting this server about twice a week I have also been spending time everyday adding to my filter files The server is a dual Xeon 2.4Ghz, 533 frontside bus with an Intel SATA raid card running Raid 10 It has about 100 small web site that do not get much traffic My goal is to reduce management time of the machine and to stabilize it so the need to reboot it is lessened I am prepared to put in a dual Xeon 3.4GH, etc but also want to make sure that I do not overkill Harry Vanderzand inTown Internet Computer Services 11 Belmont Ave. W. Kitchener, ON N2M 1L2 519-741-1222 Did you know we offer: - Province wide dial-up and high speed internet access - Web accessible email with anti-spam\antivirus protection - Computer hardware sales and service - Experienced website developers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Markus Gufler Sent: Wednesday, September 08, 2004 11:22 AM To: [EMAIL PROTECTED] Subject: RE: [Declude.JunkMail] Test needed along with sniffer I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the "spamtrap" and hope that sniffer and surbl will do this. Do I even need surbl? Do you have so much workload on your mailserver that you need to downsize your spam-filter to one or two tests? Maybe http://www2.spamchk.com/public.htm will give you some answer. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [Declude.JunkMail] Test needed along with sniffer
Harry, We have a utility to let you know how well a specific test does in our log parser (DLAnalyzer). The test is called the Test Breakdown Summary Report. Essentially you can pick a certain test(s) and see which other tests fail along with them. This report has helped us eliminate tests that performed the same as other tests. For example you can configure the report to summarize messages that failed Sniffer. It will than show you what other tests failed on messages that also failed Sniffer. You can get more granular by even excluding tests. For example: Show me which tests were triggered in conjunction with Sniffer, but did not fail XBL. Below is the link for a sample output from this report. http://www.invariantsystems.com/dlanalyzer/testsamples/TestSummaryBreakdownR eport.html In the above report you can see that out of all messages that failed the weight30 test 85% of them also failed SPAMCOP and 63% failed XBL.. Darrell Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, MRTG Integration, and Log Parsers. Harry Vanderzand writes: I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the spamtrap and hope that sniffer and surbl will do this. Do I even need surbl? Any advice in this matter would be greatly appreciated. Thanks in advance Harry Vanderzand inTown Internet Computer Services --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Test needed along with sniffer
Title: Message thank you Matt, I am running 179i16 so I may have another issue at hand here I have 42k myfilter file with every entry set to anywhere which essentially does a similar thing that surbl is doing. I mine the web info from them manually everyday. I do it on my own account as my account attracts a tremendous amount of spam I guess because it has been around for 10 years. Whatever gets through to it after declude has been going into my filter file I have surbl running with its 35k file I have today eliminated my filter file and will likely eliminate surbl once I get the full version of sniffer going. So far I see no more going through as it is likely that surbl has been better at that process than me. I am starting to realize that these body filters are expensive in cpu cycles I will share what I learn from all this I appreciate your assistance. Harry Vanderzand inTown Internet Computer Services 11 Belmont Ave. W.Kitchener, ONN2M 1L2519-741-1222Did you know we offer: - Province wide dial-up and high speed internet access - Web accessible email with anti-spam\antivirus protection- Computer hardware sales and service- Experienced website developers -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of MattSent: Wednesday, September 08, 2004 3:56 PMTo: [EMAIL PROTECTED]Subject: Re: [Declude.JunkMail] Test needed along with snifferHarry,Sniffer is a great addition to any Declude setup, however your issues are not due to just simply the size of your processors. We run a dual 1 GHz PIII system with RAID 5 and 5x10K Cheetahs, and we've managed to exceed 90,000 messages a day with dual virus scanners, and we could handle a bit more still. My thought is that you are either running a ton of BODY filters, a very slow virus scanner/scanners, or you are experiencing some form of I/O limitation. The idle processes also suggest that maybe there is an issue and an upgrade to a more recent version of Declude such as 1.79 or an interim release thereafter would be a good idea and most around here run them.You should be able to minimally do 10 times your current volume, so keep looking and keep describing your environment and a solution will likely come along.MattHarry Vanderzand wrote: I am getting service timeouts due mostly to all the declude instances of traffic volume I handle about 2 messages a day, most of them during business hours I find that I accumulate declude processes that have consumed up to a minute of cpu time only to be idle and just sit there This also causes accumulated memory to be consumed I have been rebooting this server about twice a week I have also been spending time everyday adding to my filter files The server is a dual Xeon 2.4Ghz, 533 frontside bus with an Intel SATA raid card running Raid 10 It has about 100 small web site that do not get much traffic My goal is to reduce management time of the machine and to stabilize it so the need to reboot it is lessened I am prepared to put in a dual Xeon 3.4GH, etc but also want to make sure that I do not overkill Harry Vanderzand inTown Internet Computer Services 11 Belmont Ave. W. Kitchener, ON N2M 1L2 519-741-1222 Did you know we offer: - Province wide dial-up and high speed internet access - Web accessible email with anti-spam\antivirus protection - Computer hardware sales and service - Experienced website developers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Markus Gufler Sent: Wednesday, September 08, 2004 11:22 AM To: [EMAIL PROTECTED] Subject: RE: [Declude.JunkMail] Test needed along with sniffer I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the "spamtrap" and hope that sniffer and surbl will do this. Do I even need surbl? Do you have so much workload on your mailserver that you need to downsize your spam-filter to one or two tests? Maybe http://www2.spamchk.com/public.htm will give you some answer. Markus --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
RE: [Declude.JunkMail] Test needed along with sniffer
Thank you very much. I will absorb this and share what I learn -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Wednesday, September 08, 2004 2:00 PM To: Harry Vanderzand Cc: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] Test needed along with sniffer On Wednesday, September 8, 2004, 11:13:18 AM, Harry wrote: HV I am testing sniffer right now and wonder if I need to run all the HV other tests along side it. Well, you can probably get by without the other tests, but since you have Declude it would be MUCH better if you keep the other tests in place. Declude's strength is that it allows you to aggregate a variety of tests for greater accuracy. Sniffer is very, very good, but you will certainly see some benefit by using it along with other tests. HV I am trying to reduce my daily workload of analyzing the spamtrap HV and hope that sniffer and surbl will do this. Sniffer is perfect for that - particularly if you share your spamtrap data with us. Put another way, if you allow us to use your spamtrap then we will be taking over this work for you. All we need is POP3 account information and some details on how your spamtrap was formed so that we can properly classify it in our SPHUD (Spam Processing Heads Up Display). HV Do I even need surbl? Probably not. One of the AI elements in our robots crossreferences incoming spamtrap data with SURBL and other tests. More often than not we have the domain tagged before we see it in SURBL, and if we don't we grab it quickly. HV Any advice in this matter would be greatly appreciated. I recommend reviewing the Spam Test Quality Analysis: http://www2.spamchk.com/public.html You can use this to help tune your Declude configuration. I recommend applying the forumula: W = (a^2)100 Where (W) is the individual test weight (magnitude) based on test accuracy and (a) is the accuracy measured in the analysis (SA = spam-test accuracy, HA = ham-test accuracy). [ Regarding (magnitude), ham tests generate negative weights and spam tests generate positive weights. W will always be a positive value, so if you use an HA value for (a) then you will want to apply a negative W as your weight in Declude. ] For example, SNIFFER SA = 0.95, so W = ((0.95)^2)*100 = 90.25, Weight = 90. FIVETEN-SRC SA = 0.59, so W = ((0.59)^2)*100 = 34.81, Weight = 35. NOLEGITCONTENT HA=0.38, so W = ((0.38)^2)*100 = 14.44, Weight = -14 -- This test is measured when the test does not fail, so -14 must go in second weight column, not the first. If you use this analysis you should have your hold weight at or about 100. If you set your hold weight lower than 100, you will capture more spam at the risk of more false positives. If you set your hold weight higher than 100 you will have fewer false positives and more spam. !! This is research in progress - these formulas appear to work very well in preliminary testing. If you are already happy with your weighting system then you should probably stick with that until this theory has been tested further. !! We are developing a utility to do this work automatically. In the mean time, you can go through your test weights manually. You shouldn't have to do this frequently. Hope this helps, _M --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Test needed along with sniffer
Thank you, I will try the report out. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of support Sent: Wednesday, September 08, 2004 4:06 PM To: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] Test needed along with sniffer Harry, We have a utility to let you know how well a specific test does in our log parser (DLAnalyzer). The test is called the Test Breakdown Summary Report. Essentially you can pick a certain test(s) and see which other tests fail along with them. This report has helped us eliminate tests that performed the same as other tests. For example you can configure the report to summarize messages that failed Sniffer. It will than show you what other tests failed on messages that also failed Sniffer. You can get more granular by even excluding tests. For example: Show me which tests were triggered in conjunction with Sniffer, but did not fail XBL. Below is the link for a sample output from this report. http://www.invariantsystems.com/dlanalyzer/testsamples/TestSum maryBreakdownR eport.html In the above report you can see that out of all messages that failed the weight30 test 85% of them also failed SPAMCOP and 63% failed XBL.. Darrell -- -- Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, MRTG Integration, and Log Parsers. Harry Vanderzand writes: I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the spamtrap and hope that sniffer and surbl will do this. Do I even need surbl? Any advice in this matter would be greatly appreciated. Thanks in advance Harry Vanderzand inTown Internet Computer Services --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Test needed along with sniffer
I don't know if your filters have a SKIPIFWEIGHT line. You can add a SKIPIFWEIGHT that will bypass the filters that enter that filter with a high spam weight. This should get you to bypass lots of e-mail. This probably causes me to skip 75-80% of the most obvious spam. I also have a TESTSFAILED END line for items that are psuedo-whitelisted from friendlier sites. This probably forces the body filters to be skipped on about 7-8% of the mostly non-spam messages. This leaves the battleground of about 10 to 15% of the messages that need to have body filters applied. I also put my body filters last in the global.cfg. So the quicker HELO/MAILFROM/SUBJECT/COUNTRY filters are run first. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 09/08/04 04:16PM thank you Matt, I am running 179i16 so I may have another issue at hand here I have 42k myfilter file with every entry set to anywhere which essentially does a similar thing that surbl is doing. I mine the web info from them manually everyday. I do it on my own account as my account attracts a tremendous amount of spam I guess because it has been around for 10 years. Whatever gets through to it after declude has been going into my filter file I have surbl running with its 35k file I have today eliminated my filter file and will likely eliminate surbl once I get the full version of sniffer going. So far I see no more going through as it is likely that surbl has been better at that process than me. I am starting to realize that these body filters are expensive in cpu cycles I will share what I learn from all this I appreciate your assistance. Harry Vanderzand inTown Internet Computer Services 11 Belmont Ave. W. Kitchener, ON N2M 1L2 519-741-1222 Did you know we offer: - Province wide dial-up and high speed internet access - Web accessible email with anti-spam\antivirus protection - Computer hardware sales and service - Experienced website developers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Wednesday, September 08, 2004 3:56 PM To: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] Test needed along with sniffer Harry, Sniffer is a great addition to any Declude setup, however your issues are not due to just simply the size of your processors. We run a dual 1 GHz PIII system with RAID 5 and 5x10K Cheetahs, and we've managed to exceed 90,000 messages a day with dual virus scanners, and we could handle a bit more still. My thought is that you are either running a ton of BODY filters, a very slow virus scanner/scanners, or you are experiencing some form of I/O limitation. The idle processes also suggest that maybe there is an issue and an upgrade to a more recent version of Declude such as 1.79 or an interim release thereafter would be a good idea and most around here run them. You should be able to minimally do 10 times your current volume, so keep looking and keep describing your environment and a solution will likely come along. Matt Harry Vanderzand wrote: I am getting service timeouts due mostly to all the declude instances of traffic volume I handle about 2 messages a day, most of them during business hours I find that I accumulate declude processes that have consumed up to a minute of cpu time only to be idle and just sit there This also causes accumulated memory to be consumed I have been rebooting this server about twice a week I have also been spending time everyday adding to my filter files The server is a dual Xeon 2.4Ghz, 533 frontside bus with an Intel SATA raid card running Raid 10 It has about 100 small web site that do not get much traffic My goal is to reduce management time of the machine and to stabilize it so the need to reboot it is lessened I am prepared to put in a dual Xeon 3.4GH, etc but also want to make sure that I do not overkill Harry Vanderzand inTown Internet Computer Services 11 Belmont Ave. W. Kitchener, ON N2M 1L2 519-741-1222 Did you know we offer: - Province wide dial-up and high speed internet access - Web accessible email with anti-spam\antivirus protection - Computer hardware sales and service - Experienced website developers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Markus Gufler Sent: Wednesday, September 08, 2004 11:22 AM To: [EMAIL PROTECTED] Subject: RE: [Declude.JunkMail] Test needed along with sniffer I am testing sniffer right now and wonder if I need to run all the other tests along side it. I am trying to reduce my daily workload of analyzing the spamtrap and hope that sniffer and surbl will do this. Do I even need surbl? Do you have so much workload on your mailserver that you need to downsize your spam-filter to one or two tests? Maybe http://www2.spamchk.com/public.htm will give you some answer. Markus --- [This E-mail
Re: [Declude.JunkMail] Test needed along with sniffer
Harry, I use SURBL myself in addition to 85 other filter files, for a total of 265 KB of filters. Probably only 20% of them are BODY filter lines though, and I don't think I have any ANYWHERE filters in use. I consider our installation to be heavy, but I have spent a lot of time making it efficient. I think what you should do is tier your spam blocking by weight. We operate a Hold and a Drop range, and when something hits the Drop weight we stop processing filters on it. Over 80% of the spam never runs our custom filters and that has saved us an enormous amount of CPU cycles. You would do this with the SKIPIFWEIGHT setting in the top of every custom filter file. We Hold starting at a score of 10 (mostly 13 though) and Drop at a score of 25. We manage to get 98% of the spam to land in our Drop range which we don't review at all. Our false positive rate in the Drop range is far less than 1 in 10,000, and typically results from widely blacklisted sources that no one complains about. I am only aware of about 3 FP's to land in this range over the last year. More importantly, it allows us to focus on the 2% that lands in our Hold range where we typically find about 2 to 3 FP's per 100 messages that land in there, though most of that is what we consider to be legitimate advertising or newsletters from mixed sources. I highly recommend that you focus on adding SKIPIFWEIGHT to your filters and tiering your scoring and actions appropriately. It is generally safe to toss what scores 3 times your hold weight, though some filter architectures can enhance false positives and it is important to limit incidences where the same FP issue can trip multiple filters. Matt Harry Vanderzand wrote: Message thank you Matt, I am running 179i16 so I may have another issue at hand here I have 42k myfilter file with every entry set to anywhere which essentially does a similar thing that surbl is doing. I mine the web info from them manually everyday. I do it on my own account as my account attracts a tremendous amount of spam I guess because it has been around for 10 years. Whatever gets through to it after declude has been going into my filter file I have surbl running with its 35k file I have today eliminated my filter file and will likely eliminate surbl once I get the full version of sniffer going. So far I see no more going through as it is likely that surbl has been better at that process than me. I am starting to realize that these body filters are expensive in cpu cycles I will share what I learn from all this I appreciate your assistance. Harry Vanderzand inTown Internet Computer Services 11 Belmont Ave. W. Kitchener, ON N2M 1L2 519-741-1222 Did you know we offer: - Province wide dial-up and high speed internet access - Web accessible email with anti-spam\antivirus protection - Computer hardware sales and service - Experienced website developers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Wednesday, September 08, 2004 3:56 PM To: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] Test needed along with sniffer Harry, Sniffer is a great addition to any Declude setup, however your issues are not due to just simply the size of your processors. We run a dual 1 GHz PIII system with RAID 5 and 5x10K Cheetahs, and we've managed to exceed 90,000 messages a day with dual virus scanners, and we could handle a bit more still. My thought is that you are either running a ton of BODY filters, a very slow virus scanner/scanners, or you are experiencing some form of I/O limitation. The idle processes also suggest that maybe there is an issue and an upgrade to a more recent version of Declude such as 1.79 or an interim release thereafter would be a good idea and most around here run them. You should be able to minimally do 10 times your current volume, so keep looking and keep describing your environment and a solution will likely come along. Matt Harry Vanderzand wrote: I am getting service timeouts due mostly to all the declude instances of traffic volume I handle about 2 messages a day, most of them during business hours I find that I accumulate declude processes that have consumed up to a minute of cpu time only to be idle and just sit there This also causes accumulated memory to be consumed I have been rebooting this server about twice a week I have also been spending time everyday adding to my filter files The server is a dual Xeon 2.4Ghz, 533 frontside bus with an Intel SATA raid card running Raid 10 It has about 100 small web site that do not get much traffic My goal is to reduce management time of the machine and to stabilize it so the need to reboot it is lessened I am prepared to put in a dual Xeon 3.4GH, etc but also want to make sure that I do not overkill Harry Vanderzand inTown Internet Computer