Pete,
I was judging based on the size of our Hold range which scores from 10-24. On Monday that was 1.86% of total traffic, but on Tuesday that was 2.83%. Message volume was hardly different. Other notables were that on Monday, Sniffer hit 77.27% of all E-mail but on Tuesday it hit 74.53% (both exclude Gray hits). Our overall spam percentage is about 82% on Monday and 81% on Tuesday. I did also see a drop in XBL hits which are primarily zombies from 38.14% to 34.93%. I've always found static spammers to be much more problematic because they lack many spammy patterns, and it could be that there was a wave of them that came online yesterday which could account for the difference.
I don't want to make a huge deal out of this, but I noted the drop in size from one rulebase to another and thought that might be significant, and I like to be aware of what is going on. In reality though the difference in percentages in our Hold file meant manually reviewing 50% more E-mails, or about 500 extra messages. With everything else consistent, I figured it was worth a post just to check.
I do recall an old posting where you indicated that you were going to drop the expiration down to 5 days under a certain number of hits. My thought there is that while it does present some savings in processing, it might make more sense to do a 7-8 day expiration in order to help catch spammers that are on weekly schedules, primarily lower volume niche spammers. Unfortunately I can't compare my current results accurately to the pre-change data because the makeup of my traffic has changed significantly over that time frame.
Another possibility is that our Chinese language spam might have been extra heavy. I've brought in much more of that recently from a couple different clients and it regularly scores low, probably because it's difficult to determine if most of it is spam. I do know that Sniffer doesn't do nearly as well with this stuff. I've noticed that these guys are spamming mostly during Chinese business hours, and they might have been extra light on Monday due to the lag in hours coming from a weekend. If you are interested in getting these caught messages forwarded to you in an automated fashion for study or for potential inclusion, just let me know. I also have a filter set up for Russian language E-mail, but it is not nearly as high in volume (now).
Regarding when I saw the changes in the rule base, I was pulling an all-nighter for server administration and noticed this around 5 a.m. when I ran the stats program on my Declude logs. The renamed 'old' rulebase was just over 4 MB while the active one was 4.7 MB, then at about noon I noticed it was about 4.3 MB, and now it's back up over 4.7 MB (1,000 KB = 1 MB in these stats if that matters).
I haven't yet upgraded to the most recent release, I'm still on the prior beta. I'll probably do that this evening. I tend to wait on upgrades until there has been enough time for bugs to surface unless I am already looking for a fix. I'm sure that the extra verification of the rulebase will help prevent the potential of problems, and I guess this has the possibility of being caused by a bit of corrupted data, though that's probably reaching.
Again, regardless if there was a blip, Sniffer still does a wonderful job of tagging lots and lots of E-mail, just not quite as much as the day before.
Thanks,
Matt
Pete McNeil wrote:
At 12:57 PM 5/19/2004, you wrote:
Pete,
I noted late last night that my rulebase grew by 700 KB over the size of the previous one that was archived on my machine, and also the hits for some of the tests were noticeably lower and I had a definite increase in the number of messages that scored in my Hold range (instead of scoring higher and landing in Drop). This morning though the size of my rulebase again dropped by about 450 KB.
I was just wondering if this might have been a hiccup with a bad compilation or maybe you were testing something out?
We didn't have anything under test that would alter the rulebases. I'm going to dig through the logs and see if there's anything I can identify.
If the rulebase was corrupted in any way you would have been able to detect that with the latest snf2check utility.
It's not unusual for ruelbase sizes to change by as much as 20%. The system is constantly activating and deactivating rules based on new log files that are reported. Currently a significant change might occur once per day - though we are working on new analysis engines that will permit more frequent rule strength adjustments.
For example, we might add 300-900 rules over the course of a day - then have that many (or more) removed when the new rule strength numbers are calculated.
Another factor that impacts rulebase size is the content of the rules. The folding process is not deterministic so it is possible for a few rule changes to significantly alter the way the rulebase file is folded. This is less likely to be the change but it is possible.
What was the date on the archive you used to compare sizes?
_M
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
-- ===================================================== MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =====================================================
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
