Re: Fake MX
On Wed, 8 Dec 2010, Matt wrote: Anyone using the Fake MX trick? http://www.webhostingtalk.com/wiki/Fake_MX Is it safe to use a fake high and low mx? At my last company, I found it very useful to setup the high MX's to use a greylist. I would not use a low MX for this. It was very effective at inhibiting spam -- nearly 70% of inbound spam hit the greylist first. Of that, very little (maybe 2-3%) ever retried, and therefore made it past the greylisting host. In addition to screening out a lot of crap right off the bat, it reduced the load on SA considerably. I highly recommend it. But again, not the low MX. You'd be playing with fire there. -- Jon Trulson | A828 C19D A087 F20B DFED mailto:j...@radscan.com | 67C9 6F32 31AB E647 B345 What can be asserted without evidence, can also be dismissed without evidence. -- Christopher Hitchens
Re: Hostkarma White list Updated and Improved
On Mon, 5 Oct 2009, Marc Perkel wrote: John Hardin wrote: On Mon, 5 Oct 2009, Marc Perkel wrote: Our white list is supposed to be a source of pure good email. So if spam comes for any of the white listed IPs then it's an error. Whose? Yours or theirs? Meaning: is a single spam reason for an IP to be dropped from the hostkarma whitelist? It depends on what kind of spam it is. If it is a virus generated spam - then yes. If it's a spam determined by message content - no. Sorry if I missed this in the thread, but how do you determine whether a spam originates from a bot-net vs. a 'lone wolf'? -- I drank what? | Jon Trulson -Socrates | mailto:j...@radscan.com | A828 C19D A087 F20B DFED | 67C9 6F32 31AB E647 B345
Re: more habeas spam
On Wed, 7 Jan 2009, Anthony Peacock wrote: LuKreme wrote: On 6-Jan-2009, at 08:51, Greg Troxel wrote: I realize that HABEAS_ACCREDITED_SOI has or had a reasonable ruleqa value. But, I wonder if SA should apply higher standards than that, and not give negative scores to databases that don't behave reasonably. This has been brought up on the list in the past (there was a long thread on it last February). The best suggestion I saw in that thread was [...] was something quite different from what it had been under her stewardship. I zeroed the scores for all of these rules about a year ago. They were only hitting on SPAM emails and pushing them into the FN range. I second that - habeas stopped being useful a long time ago (IMO of course :). Just zero them out. -- Happy cheese in fear | Jon Trulson against oppressor, rebel!| mailto:j...@radscan.com Brocolli, hostage. -Unknown| 4E2A 697F 66D6 7918 B684 | FEB6 4E98 16C1 25F8 A291
Re: more habeas spam
On Fri, 9 Jan 2009, John Hardin wrote: On Fri, 9 Jan 2009, Jon Trulson wrote: On Wed, 7 Jan 2009, Anthony Peacock wrote: I zeroed the scores for all of these rules about a year ago. They were only hitting on SPAM emails and pushing them into the FN range. I second that - habeas stopped being useful a long time ago (IMO of course :). Just zero them out. Erm. If they're hitting on nothing but spam, doesn't that mean you should assign them a *positive* score? ;) I didn't say they hit on nothing *but* spam :) I really have no idea how much ham they hit, but I sure noticed it when spam was allowed through because of it. So I zero'd them out, and haven't missed them at all. -- Happy cheese in fear | Jon Trulson against oppressor, rebel!| mailto:j...@radscan.com Brocolli, hostage. -Unknown| 4E2A 697F 66D6 7918 B684 | FEB6 4E98 16C1 25F8 A291
Re: Fake MX Record(s) Trick
On Mon, 23 Jun 2008, Marc Perkel wrote: Marc Ferguson wrote: Hi, I'm a linux noob and a spam assassin noob so please reply in simplified language. Thanks. I saw on the wiki a trick to use fake mx records in order to weed out spam (http://wiki.apache.org/spamassassin/OtherTricks). I'm using Evolution at home and on my laptop and I have the spamassassin plugin so I'm constantly clicking the junk icon. I have access to my shared web hosting account and I sure do get TONS of spam. I'm a bit confused as to how to implement it though. My web host uses WHM so my form looks something like this: digitalalias.net http://digitalalias.net 14400 IN MX 0 digitalalias.net http://digitalalias.net What is 14400, I'm guessing a port of some kind. Besides that the wiki suggests that my first fake mx record should be set at 10, then my real mx record at 20, and then another fake one at 30. Why is this since my current mx record is set to 0? fake0.example.com http://fake0.example.com 10 realmx.example.com http://realmx.example.com 20 fake1.example.com http://fake1.example.com 30 Hi Marc, I'm the guy who invented the trick and yes it does work. I'm running it with No you aren't. more that 4000 domains and it gets rid of more than half my spam without having to use spamassassin. I use SA too but it's very expensive to run and anything that reduces it will cut your server load. I'm also providing a public server to harvest fake MX info to help build my blacklist. You can use this host for your fake high numbered MX. (Not a low numbered MX though) Que the spamvertising... mail.yourdomain.com 10 tarbaby.junkemailfilter.com 20 -- Happy cheese in fear | Jon Trulson against oppressor, rebel!| mailto:[EMAIL PROTECTED] Brocolli, hostage. -Unknown| #include std/disclaimer.h
Re: [OT] Volume of mail thru SpamAssassin.
On Tue, 8 Jan 2008, Reg Clemens wrote: This is somewhat off topic, but I would like to be able to measure the amount of mail that comes into my mail server each day. I don't think that Sendmail has such an option, But since SpamAssassin is a series of scripts, it would seem possible to do the count there. Has anyone done anything like this? Any suggestions on how to do it? Any other way to get the count? man mailstats -- Happy cheese in fear | Jon Trulson against oppressor, rebel!| mailto:[EMAIL PROTECTED] Brocolli, hostage. -Unknown| #include std/disclaimer.h
Re: Bit OT but it's about SPAM
On Wed, 17 Oct 2007, John Rudd wrote: Bart Schaefer wrote: On 10/17/07, Tom Ray [EMAIL PROTECTED] wrote: I just thought if anyone hasn't read it yet, this article might be interesting to many of you. According to this report SPAM has now reached being 95% of all email. This is hyperbole. What it really means is that 95% of the mail processed by someone's commercial spam filter has been classified, possibly incorrectly, as spam. The rates are much lower (though still too high for comfort) if false positives are accounted for. See, for example: http://www.bcs.org/server.php?show=conWebDoc.14617 My observation, both at work and at home, is that 95% is pretty close to true. Same here. At home about 97% (~1200 rejected a day at mta with scores above 15, no less. And this is at home!). At work we fluctuate between 94-96% pure (absolutely pure!) spam. I definitely love my spamassassins :) -- Happy cheese in fear | Jon Trulson against oppressor, rebel!| mailto:[EMAIL PROTECTED] Brocolli, hostage. -Unknown| #include std/disclaimer.h
Re: FW: List of 700,000 IP addresses of virus infected computers
On Wed, 12 Sep 2007, Luis Hernán Otegui wrote: 2007/9/12, Jon Trulson [EMAIL PROTECTED]: On Wed, 12 Sep 2007, Jason Bertoch wrote: On Tuesday, September 11, 2007 7:07 PM Marc Perkel wrote: The details are a little to complex for this forum ... OK - had quite a few trolls here who seem to be hostile to my breakthroughs so I wasn't that motivated to post information. Is there any chance we can get a moderator on this, please? This is clearly not a SA topic and I'm weary of insults, flames, and advertisements from Marc. FWIW, +1 -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta OK, count me in... Be careful if you agree with others and I :) I too received the lovely 'I've added you to my blacklist' email from our buddy Marc. So be warned, you might be added too! :) If he's actually talking about this magical blacklist he's trying to sell, that should give some people pause about actually using it in real life :) -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: FW: List of 700,000 IP addresses of virus infected computers
On Wed, 12 Sep 2007, Jason Bertoch wrote: On Tuesday, September 11, 2007 7:07 PM Marc Perkel wrote: The details are a little to complex for this forum ... OK - had quite a few trolls here who seem to be hostile to my breakthroughs so I wasn't that motivated to post information. Is there any chance we can get a moderator on this, please? This is clearly not a SA topic and I'm weary of insults, flames, and advertisements from Marc. FWIW, +1 -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: Question - How many of you run ALL your email through SA?
On Mon, 20 Aug 2007, Duane Hill wrote: On Mon, 20 Aug 2007 at 16:24 -0600, [EMAIL PROTECTED] confabulated: On Fri, 17 Aug 2007, Eric A. Hall wrote: On 8/16/2007 12:39 PM, Marc Perkel wrote: OK - it's interesting that of all of you who responded this is the only person who is doing it right. I have to say that I'm somewhat surprised [...] Most blacklists I know of that have gone away in the past set DNS to return 127.0.0.2 to ALL requests that came in. Most of the email lists I'm on received posts by other list members with reguards to the list going away. I would speculate that was the reason your messages started tagging as spam. One such list I remember was ordb.org. Yes, ordb. Knew it was something like that. It may be true that they posted something to a list - unfortunately, I was not subscribed. Nonetheless, we won't do that again. -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: Question - How many of you run ALL your email through SA?
On Mon, 20 Aug 2007, David B Funk wrote: On Mon, 20 Aug 2007, Duane Hill wrote: On Mon, 20 Aug 2007 at 16:24 -0600, [EMAIL PROTECTED] confabulated: [snip..] I have to second that... In the early days when spammers were just getting started, we started using some RBL's at the MTA level. ORBS was one I believe. Then they went away and started tagging everything as spam, and of course we started rejecting everything. Lesson learned - we will not depend on any external RBL as an absolute pass/fail test ever again :) We use greylisting on the secondary MX's, but everything goes through SA eventually before entering our internal mail system. Works great. Most blacklists I know of that have gone away in the past set DNS to return 127.0.0.2 to ALL requests that came in. Most of the email lists I'm on received posts by other list members with reguards to the list going away. I would speculate that was the reason your messages started tagging as spam. One such list I remember was ordb.org. ordb.orgRIP 12/31/2006 dorkslayers.com RIP 9/15/2003 osirusoft.com RIP 8/20/2003 orbz.orgRIP 3/25/2002 orbs.orgRIP 6/3/2001 And that's just from this millenium. ;) Returning FP to ALL requests is the fastest way to wake up brain-damaged sites that don't get the clue. ordb.org, osirusoft.com, orbs.org - those were ones we used IIRC. Guess we didn't have a clue then. As mentioned earlier, for our setup anyway, it is unwise to pin pass/fail on RBL's. They can be wrong, or go away. -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: Question - How many of you run ALL your email through SA?
On Fri, 17 Aug 2007, Eric A. Hall wrote: On 8/16/2007 12:39 PM, Marc Perkel wrote: OK - it's interesting that of all of you who responded this is the only person who is doing it right. I have to say that I'm somewhat surprised that so few people are preprocessing their email to reduce the SA load. As we all know SA is very processor and memory expensive. Personally, I'm filtering 1600 domains and I route less than 1% of incoming email through SA. SA does do a good job on the remaining 1% that I can't figure out with blacklists and whitelists and Exim tricks, but if I ran everything through SA I'd have to have a rack of dedicated SA servers. third-party blacklists are good indicators but they are not perfectly accurate. the errors make them unsuitable as a sole metric, but are by definition very good inputs for spamassassin's probability scoring systems. for those of us that can afford this approach it works very well. I'm sorry you can't, but that's not our fault. I have to second that... In the early days when spammers were just getting started, we started using some RBL's at the MTA level. ORBS was one I believe. Then they went away and started tagging everything as spam, and of course we started rejecting everything. Lesson learned - we will not depend on any external RBL as an absolute pass/fail test ever again :) We use greylisting on the secondary MX's, but everything goes through SA eventually before entering our internal mail system. Works great. -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: OT Question
On Sat, 2 Dec 2006, Nigel Frankcom wrote: Hey all, Did a botnet fall over or am I just lucky? spam has dropped dramatically here ~80% down. Not that I'm complaining - just curious is anyone else is seeing the same. Huh... I too have noticed a significant drop since yesterday's stock onslaught. Without hard data available at the moment, I'd guess we are seeing a less than a third of what we were getting 24hrs ago. -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: Bayes failure on hi, it's Somebody spam
On Thu, 16 Nov 2006, Bart Schaefer wrote: It looks to me as if the recent spate of pump'n'dump spams are deliberately crafted to avoid being Bayes-learned by spamassassin. In spite of all having different subject lines and senders and other minor differences, once you've learned one of them sa-learn ignores all the rest -- and they all still get a BAYES_00 score for me. I thought I had a pretty good understanding of how SA's Bayes training worked, but this is pretty clearly confusing it somehow. Hmm, that has not been my experience at all... Bayes (99) is still catching every one for me. There may be something else going wrong with your setup - no idea what offhand though, sorry. -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: Bayesian scores
On Thu, 9 Nov 2006, Daryl C. W. O'Shea wrote: Steve Ingraham wrote: I have already decreased the Bayes_50_Body rule from 5.0 to 2.5. I don't want to decrease the scores with every Bayes rule because I think I will start seeing some true spam delivered because it did not score high. Any ideas? Don't screw with the bayes scoring that drastically? 5.0, even 2.5, for BAYES_50 is a little excessive considering that it basically means bayes has no idea if the message is ham or spam. These are the default 3.1 scores: score BAYES_00 0.0001 0.0001 -2.312 -2.599 score BAYES_05 0.0001 0.0001 -1.110 -1.110 score BAYES_20 0.0001 0.0001 -0.740 -0.740 score BAYES_40 0.0001 0.0001 -0.185 -0.185 score BAYES_50 0.0001 0.0001 0.001 0.001 score BAYES_60 0.0001 0.0001 1.0 1.0 score BAYES_80 0.0001 0.0001 2.0 2.0 score BAYES_95 0.0001 0.0001 3.0 3.0 score BAYES_99 0.0001 0.0001 3.5 3.5 I would second that definitely. I only upped the bayes 95 and 99 rules to the pre3.0 scores - didn't mess with the others. -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: BIG increase in spam today
On Wed, 1 Nov 2006, Chris wrote: I usually come home from work to find about 60-80 spam's in my spam folder. Today upon bringing up the mailer there were over 400! Looks like a large botnet attack or something. Has anyone else noticed this? I've not finished looking at the ASN's to see where they're from, but I do notice that there are about 25-30 with the same subject in each group. I've noticed a significant uptick over the last month actually - both at home and work. At work, spam is now about 95% of all inbound mail (where it was hovering in the 75-80% range for some months). Scanning is still going ok (no overloads), and still *very few* FN's. I love bayes. Secondary MX has over 12000 hosts in the greylist, whereas it was hovering around 6-7k for the last few months. So it's definitely on the rise from where I sit. At home, I've also seen an increase - approx 150 a day from around 80-90 previously. -- Jon Trulson mailto:[EMAIL PROTECTED] #include std/disclaimer.h No Kill I -Horta
Re: Q. about spam directed towards highest MX Record?
On Fri, 29 Sep 2006, Rob McEwen (PowerView Systems) wrote: Jon Trulson said: Hehe, that is an old spammer trick... Our secondary MX is pretty much 100% spam. I implemented greylisting on the secondary which reduced spam through it by about 99% :) The secondary does not do spam scanning, it's simply store and forward. Greylisting really helps in these cases. Jon, please tell me, what portion of your overall spams attempt to comes in through this secondary MX compared to all spam that you catch which are headed to your primary MX record. THAT is what I most wanted to know. Sorry, I missed that... It's hard to gauge right now as I've been running this setup for over a year. But, before greylisting was put into effect, I would say nearly 80% of our spam came through the secondary MX - it seemed to be the prefered mode of entry into our network. Most 'dictionary' type spam entered this way as well, since the MX did not have a list of valid users - it's only intended as an emergency backup after all. I highly recommend greylisting for secondary MX systems. :) Thanks! Rob McEwen PowerView Systems -- Jon Trulson mailto:[EMAIL PROTECTED] http://radscan.com/~jon #include std/disclaimer.h No Kill I -Horta
Re: Q. about spam directed towards highest MX Record?
On Wed, 27 Sep 2006, Rob McEwen wrote: (CCing Marc Perkel because I seem to recall him knowing about this) Not that I'd ever outright block based on this one factor alone, but... Does anyone have any stats about what percentage of spam is directed towards the highest MX Record? (that is, where there is more than one MX record?) Also, has anyone ever seen ANY legit mail go to the highest MX record when no mail server failure occurred? Hehe, that is an old spammer trick... Our secondary MX is pretty much 100% spam. I implemented greylisting on the secondary which reduced spam through it by about 99% :) The secondary does not do spam scanning, it's simply store and forward. Greylisting really helps in these cases. -- Jon Trulson mailto:[EMAIL PROTECTED] http://radscan.com/~jon #include std/disclaimer.h No Kill I -Horta
Re: Bombarded by German political spam
On Sun, 15 May 2005, David B Funk wrote: Tonight our site is being bombarded by German political spam or Joe-jobbed bounce fall-out. So far it appears to all be coming from trojaned PCs. Other than the specific URLs in the messages havn't found any easily identified parts to create rules for. anybody else seeing this? Absolutely :) Several hundred so far. I wonder whether it is worth the effort to write rules for these types of things? After feeding 50 or so to Bayes, they are all getting a bayes_99 now (I setup the bayes_99 score to 5.4 when I upgraded to 3.x). So far they are being trapped... -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: more spam with SpamAssassin version 3.0.2
On Sat, 14 May 2005 [EMAIL PROTECTED] wrote: I don't think 3.0.2 is worse, just that there's more spam around lately. If I take my own stats, SA is catching a slightly higher percentage of spam in the last month to 6 weeks. The RBL's I use frontline are catching more too. From January 05 to March 05 Spam accounted for around 60% of all email in. between March and now that has risen to a shade over 65%. I do notice some stuff gets through SA, but I figure spammers can play with SA as easily as the rest of us, and consequently can find ways to get round it. Fortunately, not many seem that determined. One of the things I did when I first upgraded to the 3.x series was to increase the BAYES_99 score to the pre 3.x value of 5.4. Almost all of the 'st-0-ck' spam is cought by BAYES_99 only on our systems. This helped alot. Bayes is also doing a good job with the German political onslaught we've been seeing today :) HTH Nigel On Sat, 14 May 2005 16:35:37 +0400, Valery V. Bobrov [EMAIL PROTECTED] wrote: Hello! I upgraded to SpamAssassin version 3.0.2 from 2.64 and I noticed the amount of spam messages has been increased! What sort of problem? Yours faithfully, Valery -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory issues have forced me back to 2.64
On Sat, 6 Nov 2004, Justin Mason wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Scott writes: I did realize I had big evil running.. Which by removing that it cut my memory usage to 42MB per child.. What is the recommended replacement for big evil? Is it already part of 3.0.1? SURBL. Support for it is builtin to 3.0.x by default. All the people who are reporting massive memory usage on 3.0.x, please try *without* add-on rulesets. 42MB is still about twice the normal memory usage on an x86 platform, and that's all rules, if it's that size just after startup. FWIW, I use no custom rulesets with v3 currently. At least on my system, the memory issues had nothing to do with a custom ruleset or 3. The --max-children=1 flag to spamd has 'solved' the issue for me... Average child size is around 19-20MB, until 'the event' happens, at which point it jumps to around 320MB. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory issues have forced me back to 2.64
On Mon, 15 Nov 2004, Jon Trulson wrote: On Sat, 6 Nov 2004, Justin Mason wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The --max-children=1 flag to spamd has 'solved' the issue for me... Sorry, that should be '--max-conn-per-child=1'. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: ver 3.0 opinions
On Thu, 28 Oct 2004, Bart Schaefer wrote: On Thu, 28 Oct 2004 15:21:59 -0700, Jeff Ramsey [EMAIL PROTECTED] wrote: Is version 3 really any better at stopping spam that 2.63? [...] Using it in local only mode, though, I've found it not very different. The spams that get through 3.x that do not get through 2.6x are generally (a) those that match BAYES_99, which by itself in the default configuration is no longer a large enough score to make me happy, or True. Some spam we get is soley BAYES_99. I've bumped it back up to 5.2 (like in 2.6x). -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
On Thu, 7 Oct 2004, Jon Trulson wrote: On Thu, 7 Oct 2004, Michael Parker wrote: On Thu, Oct 07, 2004 at 10:53:30AM -0600, Jon Trulson wrote: FWIW, in our case a child would go to 320MB and just stay there until the child was terminated (even after finishing a message). We do use AWL and bayes. Is it possible to try and find the msgs that was being scanned at that point in time? If so, can you reproduce by re-processing that message? Also, if you can, do an sa-learn --dump magic shortly after the jump happens and see what it says for the last expiry atime value. Does it happen to match when you saw the memory jump? I'll give that a shot this weekend when I'll have time to try to watch for it to happen. I missed the actual message (though it was only 2.5K, rejected at MTA), but I did happen to see one of the blowups happen this morning - to 325MB. The last expiry atime did indeed correspond with the time of the blowup. FWIW, here is the output of 'sa-learn --dump magic' [ pulsar ] sa-learn --dump magic 0.000 0 3 0 non-token data: bayes db version 0.000 0 175877 0 non-token data: nspam 0.000 0 132455 0 non-token data: nham 0.000 0 149125 0 non-token data: ntokens 0.000 0 1097168279 0 non-token data: oldest atime 0.000 0 1097513975 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 1097513872 0 non-token data: last expiry atime 0.000 0 345600 0 non-token data: last expire atime delta 0.000 0 15505 0 non-token data: last expire reduction count The time 1097513872 (Mon Oct 11 10:57:52 2004) - matches when the the blowup started. The scan for this particular message lasted 125.9 seconds, and ran the CPU at 99% until the child exited. I am using '--max-conn-per-child=1' option to spamd. I am also running Perl 5.8.0, if that makes a diff. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
On Fri, 8 Oct 2004, Michael Parker wrote: On Tue, Oct 05, 2004 at 12:25:45PM -0500, Michael Parker wrote: On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. Better question. Of all the folks seeing memory issues, are you using ok_languages in your config somewhere? If not, please speak up as well. I am using 'ok_locales en'. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: SA 3.0 is eating up all my memory!!!
On Wed, 6 Oct 2004, Luis Hernán Otegui wrote: In my setup this is not an option, because I run SA as a milter, via spamass-milter. If every process has to die after the scan, it cannot pass the results of the scan to Sendmail (at least, this is what happened after I tried this option, so I kept on receiving messages like this: Oct 4 09:27:55 nahuel spamass-milter[14646]: Could not extract score from So, after all, I had to discard my precious Bayes databases, and got back to good old 2.64... How odd... I too use a spamass-milter, and this works fine for me. The spamd runs on a somewhat more beefy host. On Tue, 5 Oct 2004 15:09:50 -0500, Doug Block [EMAIL PROTECTED] wrote: I had this problem till I set the max per child option to = 1 This caused spamd to kill the process used to scan every msg once it's done. Not the best answer I know but it keeps it in check -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
On Wed, 6 Oct 2004, Michael Parker wrote: On Wed, Oct 06, 2004 at 10:19:17AM -0300, Luis Hernán Otegui wrote: In my specific case, the ponit isn't only woth the big memory usage jumps, but with SA keeping the memory, and never releasing it. Highwater marks, common in most perl applicatios, don't concern me as much as these HUGE jumps in memory that folks are seeing. Jumps that just keep chewing memory without stopping. spamd opts: -c -d -m 20 --max-conn-per-child=1 There are places in the code where we could use memory a little more efficiently, I found one yesterday in fact, and we will work on these over time. In 2.x, the fork-on-demand model allowed us to be much more liberal with our data structures. We just need to wrangle that in a little with the 3.0 pre-fork code. FWIW, in our case a child would go to 320MB and just stay there until the child was terminated (even after finishing a message). We do use AWL and bayes. Michael -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
On Thu, 7 Oct 2004, Michael Parker wrote: On Thu, Oct 07, 2004 at 10:53:30AM -0600, Jon Trulson wrote: FWIW, in our case a child would go to 320MB and just stay there until the child was terminated (even after finishing a message). We do use AWL and bayes. Is it possible to try and find the msgs that was being scanned at that point in time? If so, can you reproduce by re-processing that message? Also, if you can, do an sa-learn --dump magic shortly after the jump happens and see what it says for the last expiry atime value. Does it happen to match when you saw the memory jump? I'll give that a shot this weekend when I'll have time to try to watch for it to happen. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: 3.0 scanning delays
On Fri, 1 Oct 2004, Luis Hernán Otegui wrote: Same thing here, except that it also eats as much memory as it can... Scan times keep growing bigger and bigger in time... I saw this problem too on our scanning machine (dual Xeon HT 1GB RAM), upgraded to SA 3.0 over the weekend. After awhile (4-8 hours) it would get slower and slower (to the point the milter on the mail gateway would timeout waiting for spamd to finish a message), and then unscanned email would be delivered. I tracked it down (partially) to 3 or more of the spamd threads jumping up to around 320MB allocated RAM and staying there. Easy to suck up a gig that way. As more of the spamd children 'blew up' the slower the system became due to the increased swapping. By default each spamd child will handle 200 connections before terminating and allowing the 'master' to start a new child. After several hours, these blownup spamd's would bring the machine to it's knees. What I did was add '--max-conn-per-child=1' to the spamd start line. This causes each child to die after handling one connection. I still see the occasional 'blow up' for a spamd child, but at least now it gets released as soon as that particular child as finished scanning it's message. Since then, I haven't had any more problems - running 2 days now without requiring a manual restart to regain control of the machine. Of course this is really just a workaround. spamd really should release it's allocated mem after handling a message. I have no idea what causes a spamd to explode like that - a 'special' message that exploits some bug in spamd? You guys might try that option to spamd and see if it helps. On Thu, 30 Sep 2004 11:56:01 -0600, Shane Hickey [EMAIL PROTECTED] wrote: So, I take it that no one is seeing these weird spamd delays but me? Rats. Shane Hickey [EMAIL PROTECTED] [2004-09-29 14:11]: Howdy all. I'm running version 3.0.0 on Gentoo Linux (using the 3.0.0-r1 ebuild). The machine is a dual P3/450 and it is also running sendmail 8.12.11 and it handles mail for 20 or so domains with less than 20 users total. So, the mail volume is pretty low. I'm running spamd in the following manner: /usr/sbin/spamd -d -r /var/run/spamd/spamd.pid -u mail -x -m 10 -L I'm running spamc out of my /etc/procmailrc (with no options). What I've noticed is that after spamd has been running for a little while, it starts to take longer and longer to check each message. Here is a snippet of my times from 2.64: clean message (-104.9/5.0) for user1:8 in 0.8 seconds, 1129 bytes. clean message (-104.9/5.0) for user2:8 in 0.9 seconds, 1231 bytes. clean message (-104.9/5.0) for user1:8 in 0.8 seconds, 1231 bytes. clean message (-4.9/5.0) for user1:8 in 1.1 seconds, 1046 bytes. When I first start spamd, I see times that are very close to this. But, within 10-20 minutes, they start to climb. Here is how they look right now (I started spamd 40 minutes ago). clean message (-102.8/5.0) for user1:8 in 5.8 seconds, 1282 bytes. clean message (-5.0/5.0) for user2:8 in 41.8 seconds, 2867 bytes. clean message (-100.0/5.0) for user3:8 in 37.8 seconds, 2250 bytes. If I let spamd run for several hours, I'll see times near 200 seconds per message and it seems to keep increasing. I have always had skip_rbl_checks 1 in my local.cf. But, I've been trying to isolate what's caused this new slowness, so I've also tried to first disable razor2, dcc and pyzor and that didn't seem to make much difference. Then I set use_bayes to 0 and that seems to help a little bit, but I still see long delays. The delayed times that I show above are for this configuration: # Enable the Bayes system use_bayes 0 # Enable or disable network checks skip_rbl_checks 1 use_razor2 1 use_dcc 1 use_pyzor 1 I also tried lock_method flock and I didn't see much success their either. Anyway, I was hoping someone else had seen this behavior and or maybe someone could shed some light on what might be the cause of this? Thanks, Shane -- Shane Hickey [EMAIL PROTECTED]: Network/System Consultant GPG KeyID: 777CBF3F Key fingerprint: 254F B2AC 9939 C715 278C DA95 4109 9F69 777C BF3F Listening to: The Courtship of Birdy Numnum - The Parapalegic-Homoerotic Episode -- Shane Hickey [EMAIL PROTECTED]: Network/System Consultant GPG KeyID: 777CBF3F Key fingerprint: 254F B2AC 9939 C715 278C DA95 4109 9F69 777C BF3F Listening to: The Styrenes - Cold Meat -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
RE: SA 3.0 is eating up all my memory!!!
On Fri, 1 Oct 2004, Morris Jones wrote: I found 3.0 pushing my machine into swapping as well this afternoon -- a first for me. I stopped and restarted my smtp server and spamd, and it's back to normal for now. I'm beginning to think I might be better off running spamassassin in unique processes instead of as a daemon. The load time was never terribly bad, and they certainly can't leak. See my response in a previous thread on this problem. For kicks, try --max-conn-per-child=1 to spamd see and see if your machine will last longer :) Mine did... -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
