Re: Memory usage question
Matt Kettler wrote on Thu, 16 Sep 2004 21:43:01 -0400: > Chris S reported his spamd swelling to 45mb with a huge version of > bigevil.cf he was testing. > The latest bigevil.cf needs about 40 - 50 MB *alone*! Together with several SARE rules our spamd processes were around 90 MB lately. That's where I removed bigevil (and activated SURBL on 3.0) and that other big ruleset (don't remember the name at the moment), so that our spamd prcess is now about 50 MB. I didn't see much difference in detection on the 2.63 systems, so bigevil is simply not worth the huge memory consumption. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
RE: Memory usage question
-Original Message- From: Robert Bartlett [mailto:[EMAIL PROTECTED] Sent: Friday, September 17, 2004 10:12 AM To: users@spamassassin.apache.org Subject: RE: Memory usage question -Original Message- From: Brook Humphrey [mailto:[EMAIL PROTECTED] Sent: Friday, September 17, 2004 10:04 AM To: users@spamassassin.apache.org Subject: Re: Memory usage question On Friday 17 September 2004 07:05, Chris Santerre wrote: > Yeah, bring that 50 down a little :) Maybe 10. More memory NEVER hurt > anyone! > > Currently with BigEvil I'm running 51 megs for spamd!!! But the record on a > production server is something like 145. I think it was a crazy german ;) > > Your memory usage looks pretty normal. > > I haven't updated BE in a while. Plan on doing this afternoon. For the > remaining people using BE, "WTH is wrong with you?" :-) Chris I know it's not needed but with 3.0rc4 running big evil and a few other custom rules spamd is at 53 megs and I have 5 children processes that are each eating about 56 megs apiece. I wondered if it was true but each child shows slightly different memory usage so they are reading separately. My system has 1 gig of ram and almost 4 gigs of swap. It doesn't use the swap much though. I run my system just for me right now so no other users and it filterers out about 1000 spams a day out of a total of 2000 emails a day or so at peak. I'm also running the surbl lists on this server. I upgraded from an older spamassassin install and just haven't gotten around to cleaning out the old files yet. Hey though since I went to spamassassin 3.0 I have only had about 2 or 3 emails get through in the last 2 or 3 months. With 1000 spam emails a day that's not bad. -- -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~ `'~- Brook Humphrey Mobile PC Medic, 420 1st, Cheney, WA 99004, 509-235-9107 http://www.webmedic.net, [EMAIL PROTECTED], [EMAIL PROTECTED] Holiness unto the Lord -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~ `'~- Were do you change the count again? I keep forgetting. (I mean what file do I edit to lower m50, Im going to try m10 until we upgrade our memory) Nevermind, I found it
RE: Memory usage question
-Original Message- From: Brook Humphrey [mailto:[EMAIL PROTECTED] Sent: Friday, September 17, 2004 10:04 AM To: users@spamassassin.apache.org Subject: Re: Memory usage question On Friday 17 September 2004 07:05, Chris Santerre wrote: > Yeah, bring that 50 down a little :) Maybe 10. More memory NEVER hurt > anyone! > > Currently with BigEvil I'm running 51 megs for spamd!!! But the record on a > production server is something like 145. I think it was a crazy german ;) > > Your memory usage looks pretty normal. > > I haven't updated BE in a while. Plan on doing this afternoon. For the > remaining people using BE, "WTH is wrong with you?" :-) Chris I know it's not needed but with 3.0rc4 running big evil and a few other custom rules spamd is at 53 megs and I have 5 children processes that are each eating about 56 megs apiece. I wondered if it was true but each child shows slightly different memory usage so they are reading separately. My system has 1 gig of ram and almost 4 gigs of swap. It doesn't use the swap much though. I run my system just for me right now so no other users and it filterers out about 1000 spams a day out of a total of 2000 emails a day or so at peak. I'm also running the surbl lists on this server. I upgraded from an older spamassassin install and just haven't gotten around to cleaning out the old files yet. Hey though since I went to spamassassin 3.0 I have only had about 2 or 3 emails get through in the last 2 or 3 months. With 1000 spam emails a day that's not bad. -- -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~ `'~- Brook Humphrey Mobile PC Medic, 420 1st, Cheney, WA 99004, 509-235-9107 http://www.webmedic.net, [EMAIL PROTECTED], [EMAIL PROTECTED] Holiness unto the Lord -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~ `'~- Were do you change the count again? I keep forgetting. (I mean what file do I edit to lower m50, Im going to try m10 until we upgrade our memory)
Re: Memory usage question
On Friday 17 September 2004 07:05, Chris Santerre wrote: > Yeah, bring that 50 down a little :) Maybe 10. More memory NEVER hurt > anyone! > > Currently with BigEvil I'm running 51 megs for spamd!!! But the record on a > production server is something like 145. I think it was a crazy german ;) > > Your memory usage looks pretty normal. > > I haven't updated BE in a while. Plan on doing this afternoon. For the > remaining people using BE, "WTH is wrong with you?" :-) Chris I know it's not needed but with 3.0rc4 running big evil and a few other custom rules spamd is at 53 megs and I have 5 children processes that are each eating about 56 megs apiece. I wondered if it was true but each child shows slightly different memory usage so they are reading separately. My system has 1 gig of ram and almost 4 gigs of swap. It doesn't use the swap much though. I run my system just for me right now so no other users and it filterers out about 1000 spams a day out of a total of 2000 emails a day or so at peak. I'm also running the surbl lists on this server. I upgraded from an older spamassassin install and just haven't gotten around to cleaning out the old files yet. Hey though since I went to spamassassin 3.0 I have only had about 2 or 3 emails get through in the last 2 or 3 months. With 1000 spam emails a day that's not bad. -- -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~- Brook Humphrey Mobile PC Medic, 420 1st, Cheney, WA 99004, 509-235-9107 http://www.webmedic.net, [EMAIL PROTECTED], [EMAIL PROTECTED] Holiness unto the Lord -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-
Re: Memory usage question
On Fri, Sep 17, 2004 at 10:42:20AM -0400, Matt Kettler wrote: > Since your box has 256mb of physical ram, I'd limit it to maximum of > 256mb/15mb = 17 spamd's at the highest. I'd really suggest using something > much lower like 10 unless you add some ram. Even this seems to be dangerous (sometimes). We just had a crash of the spamd-server, seemingly by being hit with lots of maximally large mails at the same time. So a system with 1G Memory (+2GSwap) DualPentium4 simply stopped completely just by crowding its space with max 32 copies of spamd (each forking with near 50M). The system was to slow to reboot correctly and had to be 'reset' and fsck-ed. So you have to watch closely, if the system is as small as the above. Better invest in lots of memory... Stucki -- Christoph von Stuckrad * * |nickname |<[EMAIL PROTECTED]>\ Freie Universitaet Berlin |/_*|'stucki' |Tel(days):+49 30 838-75 459| Fachbereich Mathematik, EDV|\ *|if online|Tel(else):+49 30 77 39 6600| Arnimallee 2-6/14195 Berlin* * |on IRCnet|Fax(alle):+49 30 838-75454/
RE: Memory usage question
At 09:23 AM 9/17/2004, Robert Bartlett wrote: Are you using the -m parameter of spamd to limit the number of children it will spawn? I'd suggest something like -m 6 to start with. Yeah it is setup for 50: -d -c -a -m50 -u user -v -H 50 is a LOT of spamd's... even at the low-end of 15mb each that's 750mb of memory allocation. Since your box has 256mb of physical ram, I'd limit it to maximum of 256mb/15mb = 17 spamd's at the highest. I'd really suggest using something much lower like 10 unless you add some ram.
RE: Memory usage question
>-Original Message- >From: Robert Bartlett [mailto:[EMAIL PROTECTED] >Sent: Friday, September 17, 2004 9:24 AM >To: users@spamassassin.apache.org >Subject: RE: Memory usage question > > >-Original Message- >From: Matt Kettler [mailto:[EMAIL PROTECTED] >Sent: Friday, September 17, 2004 6:12 AM >To: [EMAIL PROTECTED]; users@spamassassin.apache.org >Subject: RE: Memory usage question > >At 06:56 PM 9/16/2004 -0700, Robert Bartlett wrote: >>Thanks for the reply! Here is the deal, we are currently deciding what >>we want to do next. Currently we have a Celeron 2.4 gig >system with 256 >>megs of ram and a 40 gig hdd. In the past week or so our system has >come >>to a halt, under 3 megs available, due to a bunch of emails coming in >at >>once. At one point we hit 60 emails in a span of 5 minutes. > >Are you using the -m parameter of spamd to limit the number of children >it >will spawn? I'd suggest something like -m 6 to start with. > > >Yeah it is setup for 50: > >-d -c -a -m50 -u user -v -H > >-d, --daemonizeDaemonize >-c, --create-prefs Create user preferences files >-a, --auto-whitelist, --whitelist Use auto-whitelists >-u username, --username=username Run as username >-v, --vpopmail Enable vpopmail config >-H dir Specify a different HOME directory, >path optional > Yeah, bring that 50 down a little :) Maybe 10. More memory NEVER hurt anyone! Currently with BigEvil I'm running 51 megs for spamd!!! But the record on a production server is something like 145. I think it was a crazy german ;) Your memory usage looks pretty normal. I haven't updated BE in a while. Plan on doing this afternoon. For the remaining people using BE, "WTH is wrong with you?" :-) --Chris
RE: Memory usage question
-Original Message- From: Matt Kettler [mailto:[EMAIL PROTECTED] Sent: Friday, September 17, 2004 6:12 AM To: [EMAIL PROTECTED]; users@spamassassin.apache.org Subject: RE: Memory usage question At 06:56 PM 9/16/2004 -0700, Robert Bartlett wrote: >Thanks for the reply! Here is the deal, we are currently deciding what >we want to do next. Currently we have a Celeron 2.4 gig system with 256 >megs of ram and a 40 gig hdd. In the past week or so our system has come >to a halt, under 3 megs available, due to a bunch of emails coming in at >once. At one point we hit 60 emails in a span of 5 minutes. Are you using the -m parameter of spamd to limit the number of children it will spawn? I'd suggest something like -m 6 to start with. Yeah it is setup for 50: -d -c -a -m50 -u user -v -H -d, --daemonizeDaemonize -c, --create-prefs Create user preferences files -a, --auto-whitelist, --whitelist Use auto-whitelists -u username, --username=username Run as username -v, --vpopmail Enable vpopmail config -H dir Specify a different HOME directory, path optional
RE: Memory usage question
At 06:56 PM 9/16/2004 -0700, Robert Bartlett wrote: Thanks for the reply! Here is the deal, we are currently deciding what we want to do next. Currently we have a Celeron 2.4 gig system with 256 megs of ram and a 40 gig hdd. In the past week or so our system has come to a halt, under 3 megs available, due to a bunch of emails coming in at once. At one point we hit 60 emails in a span of 5 minutes. Are you using the -m parameter of spamd to limit the number of children it will spawn? I'd suggest something like -m 6 to start with.
Re: Memory usage question
> Thanks for the reply! Here is the deal, we are currently deciding what > we want to do next. Currently we have a Celeron 2.4 gig system with 256 > megs of ram and a 40 gig hdd. In the past week or so our system has come > to a halt, under 3 megs available, due to a bunch of emails coming in at > once. At one point we hit 60 emails in a span of 5 minutes. It is a You don't sound like you have a huge mail rate. But if you are using addon rules, you may be getting in the area of doubling or more the original number of rules, so you should probably at least double the memory requirement per spamassassin process (or process driver). My suggestion is that that machine is plenty fast enough for you, but I would at least double the memory on it. If its cheap I'd take it up to a gig or more and be done with it. Loren
RE: Memory usage question
-Original Message- From: Matt Kettler [mailto:[EMAIL PROTECTED] Sent: Thursday, September 16, 2004 6:43 PM To: [EMAIL PROTECTED]; users@spamassassin.apache.org Subject: Re: Memory usage question At 09:26 PM 9/16/2004, Robert Bartlett wrote: >I remember someone saying something about memory usage per email that >spamd uses to scan? But cannot find the email, what is the estimated >amount of memory used per SA scan? I also have clamav set up Varies a lot depending on your configuration (bayes vs no bayes, add on rules, etc). If I start spamd on my system (don't normaly use it because I use MailScanner which calls the API directly) it pops up with a RSS of 26mb. I use bayes with an enlarged database size (200k tokens, instead of 150k) , and a few add-on rules. A 200k token bayes db should be about 10mb based on info in the manpage, so disabling bayes and using only stock rules could take spamd down to as little as 15mb, however, I've not got the ability to test that right now. Chris S reported his spamd swelling to 45mb with a huge version of bigevil.cf he was testing. Thanks for the reply! Here is the deal, we are currently deciding what we want to do next. Currently we have a Celeron 2.4 gig system with 256 megs of ram and a 40 gig hdd. In the past week or so our system has come to a halt, under 3 megs available, due to a bunch of emails coming in at once. At one point we hit 60 emails in a span of 5 minutes. It is a system we are "renting" at a colo. So Im heading this project up trying to decide to either rent out cabinet space and build our own systems and do it that way, or just upgrade the current system. Currently we are running Fedora Core 1 with clamav. I do not believe we have bayes running, I assume we don't since I do not know how this would be set up. We also use Rules De Jour with all rules available except Big Evil, we are using RBL. I know that when I restart spamd it shows this: 99.9 9.5 29068 24300 (24300 being RSS) I also use vpopmail for virtual domain setup.
Re: Memory usage question
At 09:26 PM 9/16/2004, Robert Bartlett wrote: I remember someone saying something about memory usage per email that spamd uses to scan? But cannot find the email, what is the estimated amount of memory used per SA scan? I also have clamav set up Varies a lot depending on your configuration (bayes vs no bayes, add on rules, etc). If I start spamd on my system (don't normaly use it because I use MailScanner which calls the API directly) it pops up with a RSS of 26mb. I use bayes with an enlarged database size (200k tokens, instead of 150k) , and a few add-on rules. A 200k token bayes db should be about 10mb based on info in the manpage, so disabling bayes and using only stock rules could take spamd down to as little as 15mb, however, I've not got the ability to test that right now. Chris S reported his spamd swelling to 45mb with a huge version of bigevil.cf he was testing.
