Re: spamd still burning CPU in 3.0.1
All, > email builder wrote: > > > > >How much email are you processing ? > > > > Well, just the other day we had an average of 48 msgs/min (max 255/min) > > get run through SA. Can't say today yet because can't run our stats > > tools until the busy hours are over cuz SA is hogging the CPU. ;) > > Hi, > > Your CPU is over loaded. At 48 a minute it should run just ok on a 2.8 > Ghz machine, much over that it's going to start having problems. On our > 2.4 Ghz (not HT) processor if I process over 35 a minute I start having > problems with load. > > I'd recommend upgrading to a dual server or perhaps putting in a second > server with round robin DNS (or if you can do it, a load balancer). > > SA is that CPU intensive, it really is. Maybe try adding RBL's in front > of the MTA to reduce the number of messages you have to scan, that's > what we do. > > Regards, > > Rick Just to top off this thread, I wanted to let all the wonderful people who offered their system stats/specs know that we added a 2nd machine that is a dedicated SA server where the only other app running is MySQL (for Bayes/AWL) and things are humming along very nicely. One server apparently just couldn't handle the load we had. Thanks again all! __ Do you Yahoo!? Yahoo! Mail - You care about security. So do we. http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
> > > Out of addled curiosity (not pointing specifically at you David) why
> has
> > > nobody mentioned the traditional "SpamAssassin is slow" mantra, "Try
> > > more memory?"
> >
> > I think because memory does not seem to be an issue for me. I have 1GB
> RAM
> > and each spamd process sits at around 34MB. I don't have any swapping at
> > all. I could throw another gig in, but that doesn't really seem to be
> the
> > problem from what I can tell. Am I overlooking something?
>
> Oh, possibly a DNS timeout on a BL that has gone away?
This wouldn't cause CPU burnage, would it? From what people have told me
thus far, this would not have any relationship to CPU resources (although of
course it'd slow down overall processing). In any case, my SA is just a
stock install of 3.0.1, so I doubt this is an issue.
> Limit the number of spamds that can be run?
Yeah, I do that. I can only get away with 5 children, otherwise the machine
grinds to a practical halt. From what I've read, a machine like this (2.8G
HT, 1G RAM) should be able to run even 30 children without much trouble,
which is why I've been scratching my head. and I've not found any answers
thus far. Hmph.
> Are you running any really big (and obsolete) rule sets like BigEvil?
> {^_^}
Nope, like I said, stock 3.0.1.
Thank you!
__
Do you Yahoo!?
Check out the new Yahoo! Front Page.
www.yahoo.com
Re: spamd still burning CPU in 3.0.1
Spamd1 - 4 handle an average of 1.5 million messages per day, 810 per minute. Each box is configured to a max child of 128, and usually hover around 70% cpu idle, and 500 megs of ram free. Very impressive. I have a single spamd box, running 3.0.1, with four 3gHz Xeons and 4-gigs of memory. It's handling about 150,000 messages a day with peaks in the 350/minute range. I also have around 70% cpu idle and 500 megs of memory free most of the time. I've never used any swap. However when we tested 3.0 on one of the live spamd boxes, even after throttling the max-child down to say 64, the cpu's are pegged, and around 500 megs into swap. Another interesting thing I noticed, when SA 2.63 is set to 128 children it only spawns them as needed. SA 3.0 likes to spawn the full number of children no matter what! How many spamd processes are alive (on average) on your 2.63 box? That's the number that I would choose to startup under 3.0. It's a different philosophy: 2.6 starts up children as needed, up to the max. So as mail comes in there is a performance hit to start these processes. 3.0 starts up however many you ask for at the beginning so that they're already running and ready to go. Since my spamd box is dedicated, I prefer the new prefork model. FWIW, I start up 40 children. Each one claims a memory size of about 43 meg. I'm running the standard rules plus 16 of the SARE rule sets. -- ___ Rick Beebe(203) 785-6416 Manager, Systems & Network Engineering FAX: (203) 785-3481 ITS-Med Production Systems[EMAIL PROTECTED] Yale University School of Medicine Suite 124, 100 Church Street South http://its.med.yale.edu New Haven, CT 06519 ___
Re: spamd still burning CPU in 3.0.1
We have a nice e-mail setup with 5 inbound mx boxes (Qmail + QmailScanner + ClamD), 4 spamd boxes, 2 outbound smtp, 1 imap/pop server, and a pq (problem queue) box that mx can re-route mail to if there is a customer issue. Every box is a Dual CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) w/ 2-4 gigs of ram. Running FreeBSD 5.2.1 Our spamd boxes are running SA 2.63. We created a spamd-beta box running 3.0 for a few e-mail boxes and LOVE the upgrades. Spamd1 - 4 handle an average of 1.5 million messages per day, 810 per minute. Each box is configured to a max child of 128, and usually hover around 70% cpu idle, and 500 megs of ram free. However when we tested 3.0 on one of the live spamd boxes, even after throttling the max-child down to say 64, the cpu's are pegged, and around 500 megs into swap. Another interesting thing I noticed, when SA 2.63 is set to 128 children it only spawns them as needed. SA 3.0 likes to spawn the full number of children no matter what! Do I stay with 2.63 (which is behind the times these days and misses too much spam) Do I add more machines? Do I wait for some memory/cpu improvements in some future version of SA. Ryan Pavely Director Research And Development Net Access Corporation Jeff Koch wrote: We have two production mailservers running SA spamd. The first handles about 5,000 incoming emails per hour, does spam filtering with SA and virus filtering with qmailscanner and forwards the filtered mail to a server handling the pop accounts. We're using SA 2.64 with Bayes, AWL, Razor and about half of the RBL's. The machine is a 2.8Ghz P4 with 1.0GB RAM and SCSI hard drive. CPU usuage runs between 25-40% and system load runs 1.50 to 2.20 with isolated spikes to 7.0. The second machine is a 2Ghz Athlon with 1.0GB RAM and an IDE drive. It does spam and virus filtering with SA 2.64 and qmailscanner and also handles POP3 sessions with vpopmail. We use Bayes, AWL, Razor and the same RBL's. It handles approx 2,500 emails per hour (with peaks of 5K emails/hour) and approx 2,000 pop3 sessions per hour (peaks of 5K pops/hour). CPU usage runs about 20% with peaks to 50% and system load averages 0.80 with peaks of 16.0. We are pretty satisfied with the above setup. We tried moving one of the servers to SA 3.0 in order to use the new MySQL Bayes features but got absolutely killed on CPU usage and system load - that lasted about a day and we reverted to 2.64. We figure that we'd have to reduce the email load on each server by 50% in order to use SA 3.0 and thereby need twice as many servers. However, we're going to wait until the SA developers take the memory and load issues seriously and fix the problem. Maybe if enough users complain they'll do some high volume production test comparisons of 3.0 with previous versions and sort out the problem. At 09:33 PM 10/27/2004, email builder wrote: > email builder wrote: > >>email builder wrote: > >>How much email are you processing ? > > > > > > Well, just the other day we had an average of 48 msgs/min (max 255/min) > get > > run > > through SA. Can't say today yet because can't run our stats tools until > the > > busy hours are over cuz SA is hogging the CPU. ;) > > Hi, > > Your CPU is over loaded. At 48 a minute it should run just ok on a 2.8 > Ghz machine, much over that it's going to start having problems. On our > 2.4 Ghz (not HT) processor if I process over 35 a minute I start having > problems with load. I have two reactions to this: 1) I like the glimmer of hope and the idea that throwing hardware at the problem can solve it 2) Throwing hardware at problems is usually avoiding fixing the *real* problem. According to other posters on this list, my load is not excessive for a modern-day 2.xGHz machine. I will have to re-read some messages, but I believe responders to my posts on the "[OT] Email Servers" thread quoted similar machine specs and higher load than me and said they did not have load problems. I'd love to hear that I am mistaken and that it's just a matter of too little hardware, but I am skeptical... > I'd recommend upgrading to a dual server or perhaps putting in a second > server with round robin DNS (or if you can do it, a load balancer). We've been thinking about a multiple-machine email solution and have been wondering about architecture. Since SA seems to be the *only* email server module that causes us grief (even amavisd-new/clamav is nicer to our machine!!), and although it seems strange not to go with a separate file server or database server machine (or to otherwise split up SMTP and IMAP, etc), I am starting to think (as you suggest) that just adding a separate SA server is going to get us the biggest performance increase. What are people's opinions and experience setting up separate/multiple SA servers? Are there any good links for reading about such setups on the wiki or anywhere else? > SA is that CPU intensive, it really is.
Re: spamd still burning CPU in 3.0.1
> Out of addled curiosity (not pointing specifically at you David) why has > nobody mentioned the traditional "SpamAssassin is slow" mantra, "Try > more memory?" I think because memory does not seem to be an issue for me. I have 1GB RAM and each spamd process sits at around 34MB. I don't have any swapping at all. I could throw another gig in, but that doesn't really seem to be the problem from what I can tell. Am I overlooking something? __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
RE: spamd still burning CPU in 3.0.1
Kurt, > I've lost track of your original post, but I do have a quick question. > > From what little I do remember of your postings, I believe that you were > running SA on FreeBSD. Sorry, no, this is a Fedora Core 2 machine > From reading several FreeBSD lists, HT is problematic, and often reduces > processor efficiency. > > Have you tried it w/o HT? Does that make a difference? Tried with FC1 w/out HT, FC1 w/ HT and FC2 w/ HT, nothing seemed to help. Thanks! __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Re: spamd still burning CPU in 3.0.1
Ayup modulo a typu I do. Don't forget I am an old troglodyte
paleocomputer type who is quite contented with a few remote ssh
logins to 60 line command line sessions. That's MUCH lighter weight
than playing with X. I do have some sanity left, ya know. Here's a
piece of top at the moment - yeah 256m is closer to the point:
===8<==
18:28:31 up 22 days, 9:16, 7 users, load average: 1.40, 2.73, 2.75
78 processes: 77 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 1.9% user 1.9% system 0.0% nice 0.0% iowait 96.1% idle
Mem: 255776k av, 150048k used, 105728k free, 0k shrd, 29424k
buff
74864k actv, 27032k in_d,3176k in_c
Swap: 1161168k av, 69772k used, 1091396k free 46476k
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
2716 jdow 16 0 1168 1168 852 R 2.3 0.4 0:00 0 top
18700 root 18 0 2404 0 S 1.5 0.0 0:31 0 sshd
1 root 15 0 104 8056 S 0.0 0.0 0:28 0 init
2 root 15 0 00 0 SW0.0 0.0 0:00 0 keventd
3 root 15 0 00 0 SW0.0 0.0 0:00 0 kapmd
4 root 34 19 00 0 SWN 0.0 0.0 0:00 0
ksoftirqd_CPU
9 root 25 0 00 0 SW0.0 0.0 0:00 0 bdflush
5 root 15 0 00 0 SW0.0 0.0 1:06 0 kswapd
6 root 15 0 00 0 SW0.0 0.0 0:00 0 kscand/DMA
7 root 15 0 00 0 SW0.0 0.0 21:03 0
kscand/Normal
8 root 15 0 00 0 SW0.0 0.0 0:00 0
kscand/HighMe
10 root 15 0 00 0 SW0.0 0.0 0:07 0 kupdated
===9<>8==
(oh look, the scissors are mating! I wonder if the babies are paperclips
or wire coathangers)
Of course, keeping Linux up these days is nothing like the 2.0.36 days
when I had a machine up for about 460 days before I took it down to
put in a second NIC card when we got DSL service here.
The machine WILL run X. It is "a trifle slow." The same machine will
probably run XP, too. It'll even be faster than the slowest XP machine
I have seen. But I'd be WAY to impatient to use it.
Of course, there is a fellow who put a functional web browser into a
Commodore 64. So while spamd might be out of the question in 256k
bringing up a capable system in 256k should be easy, lots of room.
Haven't you ever run across the HTTP server in a minipic and one
serial memory chip?
{^_-} Joanne.
- Original Message -
From: "einheit elf" <[EMAIL PROTECTED]>
To:
Sent: 2004 October, 28, Thursday 15:37
Subject: Re: spamd still burning CPU in 3.0.1
> Jdow, think about what you're saying - do you really expect us to believe
> that you could even boot redhat 9, let alone do anything useful, in 256 k
> of RAM? I was able to bring up a slackware machine with 4 MB RAM, and even
> run a web server (slowly), but that's about the practical limit, and you
> claim to be running SA in 1/16th that amount of RAM?
>
> ;)
>
> jdow said:
> > Redhat 9 does. It's rather slow. But it does get there. It's for two
users
> > only. But we're both in the 1000+ emails a day class users.
> >
> > {^_^}
> > - Original Message -
> > From: "einheit elf" <[EMAIL PROTECTED]>
> >>
> >> jdow said:
> >>
> >> > I normally run SpamAssassin (2.63) on a slow machine, a
> >> > 166MHz Pentium with only 256k of ram.
> >>
> >> Pray tell, what OS enables you to run SA with only a quarter Meg of
RAM?
> >>
> >> einheit
> >>
> >
> >
>
>
> --
>
Re: spamd still burning CPU in 3.0.1
> 89 a minute! Wow! What else do you run on that machine? (Do you run your > other email server software there or is it a dedicated SA box? Do you also > run a virus scanner for example?) Hiya, It runs FreeBSD 4.8 (with SMP kernel of course) and sendmail + SA 3.0.1 - that's it, nothing else apart from mrtg and some perl scripts I run in cron to check the sanity of running processes and generate some stats for me. It used to run SA 2.64, I honestly didn't notice any major increase in system load when I upgraded to 3.0 (which was less than a fortnight ago). We "pipe" MX records through it for customer domains that wish to be spam filtered, it works pretty well. We have another server which is identical in hardware but instead runs postfix + ClamAV. This machine handles a *lot* more mail (not everyone wants their mail piped through our SA server - but they do want it AV checked) and I actually just had a new machine arrive yesterday that I'll be building up to replace it, as that box is quite heavily loaded. I'd actually like to convert our SA server into running postfix, but as it's already running quite nicely with plain old sendmail, it's not something that I really have the need to start mucking around with. Maybe in 6-12 months when we have more domains going through it. The only times I've seen our SA server busy is when there was a network problem, it didn't receive mail for a while then it suddenly got hammered hard when the network connectivity was restored, and the queues started connecting to the box, that's the only time I've wanted more CPU & RAM, that took a fair while for it to work itself out. Cheers Gav
Re: spamd still burning CPU in 3.0.1
Jdow, think about what you're saying - do you really expect us to believe
that you could even boot redhat 9, let alone do anything useful, in 256 k
of RAM? I was able to bring up a slackware machine with 4 MB RAM, and even
run a web server (slowly), but that's about the practical limit, and you
claim to be running SA in 1/16th that amount of RAM?
;)
jdow said:
> Redhat 9 does. It's rather slow. But it does get there. It's for two users
> only. But we're both in the 1000+ emails a day class users.
>
> {^_^}
> - Original Message -
> From: "einheit elf" <[EMAIL PROTECTED]>
>>
>> jdow said:
>>
>> > I normally run SpamAssassin (2.63) on a slow machine, a
>> > 166MHz Pentium with only 256k of ram.
>>
>> Pray tell, what OS enables you to run SA with only a quarter Meg of RAM?
>>
>> einheit
>>
>
>
--
Re: spamd still burning CPU in 3.0.1
Redhat 9 does. It's rather slow. But it does get there. It's for two users
only. But we're both in the 1000+ emails a day class users.
{^_^}
- Original Message -
From: "einheit elf" <[EMAIL PROTECTED]>
>
> jdow said:
>
> > I normally run SpamAssassin (2.63) on a slow machine, a
> > 166MHz Pentium with only 256k of ram.
>
> Pray tell, what OS enables you to run SA with only a quarter Meg of RAM?
>
> einheit
>
Re: spamd still burning CPU in 3.0.1
jdow said: > I normally run SpamAssassin (2.63) on a slow machine, a > 166MHz Pentium with only 256k of ram. Pray tell, what OS enables you to run SA with only a quarter Meg of RAM? einheit
Re: spamd still burning CPU in 3.0.1
From: "David Brodbeck" <[EMAIL PROTECTED]>
> On Wed, 27 Oct 2004 21:43:16 -0700 (PDT), email builder wrote
> > Thanks. We were thinking about a NFS server, but SA concerns seemed
> > more important. If both can coexist peacefully, this may be the
> > exact same solution that we use.
>
> It seems like it'd be a good match. NFS is highly I/O intensive, but
doesn't
> use the CPU much. SA is highly CPU intensive, but does relatively little
I/O.
> Just make sure you have enough memory; you don't want SA's hunger for RAM
to
> starve the machine for disk cache space, and you sure as heck don't want
any
> swapping.
Out of addled curiosity (not pointing specifically at you David) why has
nobody mentioned the traditional "SpamAssassin is slow" mantra, "Try
more memory?" I normally run SpamAssassin (2.63) on a slow machine, a
166MHz Pentium with only 256k of ram. I also have a test install, pending
retiring the old machine, that is a 2GHz Athlon with 1 gig of ram. The
newer machine appears to run much faster than the ratio of the CPU clock
speeds would allow. Usually I get much less than a 1:1 speed improvement
when upgrading the CPU, based on past observations. (Kernel compiles are
not NEARLY 13 times as fast on the newer machine, for example.) I
attribute much of the difference to having a massive overload of memory
on the newer machine.
Now, it is a problem if a spamd balloons to 100megs or more and stays
there. It is not particularly a problem if 3.0.1 uses twice or three
times the memory of 2.63. Throw more memory at it. Memory is cheaper
than your time spent trying to work around large Bayes files and large
rule sets.
{^_-} Joanne, being controversial again. (And those going to
ApacheCon PLEASE buttonhole the geek who has this list
going through a spam filter. I've had dozens of novel
attacks come through recently. And I can't post them to
the list for the SARE people. For example a modification
of the drug rules is now needed for "m or tga ge" and
"a pp lica ti o n". Spam is food for this list, at least
as attachments.)
Re: spamd still burning CPU in 3.0.1
On Wed, 27 Oct 2004 21:43:16 -0700 (PDT), email builder wrote > Thanks. We were thinking about a NFS server, but SA concerns seemed > more important. If both can coexist peacefully, this may be the > exact same solution that we use. It seems like it'd be a good match. NFS is highly I/O intensive, but doesn't use the CPU much. SA is highly CPU intensive, but does relatively little I/O. Just make sure you have enough memory; you don't want SA's hunger for RAM to starve the machine for disk cache space, and you sure as heck don't want any swapping.
Re: spamd still burning CPU in 3.0.1
On Thu, Oct 28, 2004 at 09:54:42AM -0400, Jeff Koch wrote: > > You are correct and I apologize to the SA team. I cannot characterize the > problem as a bug - SA 3.0 is just much slower and resource intensive than > SA 2.64. If I understand you correctly you are just testing Bayes. Our > production testing involved using SA as a whole. And I again suggest that > SA 3.0 be compared against previous versions (like 2.64) in a real world > production test. Maybe the answer is to publish a cheat sheet of new > features in 3.0 that need to be turned off in order to achieve the > throughput of 2.64. > Actually, it tests most local rules (auto learning with just the BAYES_* rules is kind a boring if you know what I mean). I have done comparative tests against 2.6x and 3.0.x and at the time found 3.0.x some percentage slower but it was a percentage close to the number of extra rules 3.0.x was running at the time. Since then the number of active rules in 3.0 have been reduced. Sorry, it was a PITA to run the benchmark on 2.6x since it doesn't provide some of the features that 3.0 provides, so I doubt I would do it again. I think the majority of problems people are seeing can be solved with some tuning. Indeed, 3.0 is a slightly different animal than 2.6x was. It is better and more feature rich than any previous versions. With features come complexity and unfortunately the odd problem here or there. As developers we can not cover every possible setup, for that we must rely on users. I love to help when I can, especially if you're running SQL (prefs,bayes or awl). Feel free to hop on IRC and ask for help or ask here, or if you find a bug file a report. Come to ApacheCon. I'll be there and I won't speak for anyone else, but I bet you can find help/tips from folks there. Perhaps a BOF for SpamAssassin is in order. Michael pgpLd2zj5rPvm.pgp Description: PGP signature
Re: spamd still burning CPU in 3.0.1
To clarify - the first server handles 700 domains and the second 250. The first is only handling virus and spam filtering for incoming email while the second is doing that plus pop3 and outgoing mail. The first is also SCSI which seems to help alot - especially for qmail. Oh, also in both machines we use MySQL per-user spam preferences which puts another big load on the servers. At 04:36 AM 10/28/2004, John Andersen wrote: On Thursday 28 October 2004 12:18 am, email builder wrote: > > We have two production mailservers running SA spamd. The first handles > > about 5,000 incoming emails per hour, does spam filtering with SA and > > virus > > Can I ask you how you load balance between the two machines (obviously if > one handles 5000/hr and the other 2,500, it's not straight round robin)? I saw nothing in his post to suggest he balanced load, or even that the two servers were serving the same domains. I just took it at face value that with 3.0.1 they couldn't keep up, but falling back to 2.64 he could carry the load. -- _ John Andersen Best Regards, Jeff Koch, Intersessions
Re: spamd still burning CPU in 3.0.1
email builder wrote: >>I'd recommend upgrading to a dual server or perhaps putting in a second >>server with round robin DNS (or if you can do it, a load balancer). > > > also, what do people think about a multiple cpu machine vs more than one > machine? dumb question? (two machines always are faster than one dual-cpu > machine) I tend to prefer dual CPU machines as servers. They have the large advantage that if some badly behaved process wedges the processor, you just get one CPU running flat out, and the machine remains responsive. Now you can ameliorate this sort of thing with ulimits and such like, but nothing beats having a second CPU in the box, IMO. It also gives you redundancy in case of failure, as most dual CPU machines will run fine on 1 CPU for a while. Regarding your problems, I'm sure there must be somethin wrong - SA should not be that CPU intensive. I'm running it on a dual CPU Athlon 2000, using spamd via a sendmail milter. CPU usage for the individula spamd child processes never seems to go much above about 8-9%, and even that is momentary. After a 65 day uptime, some of my spamd processes (quickly pasted from top) look more or less like this:- PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 11257 spamd 9 0 21200 17M 5144 S 0.0 1.6 0:18 spamd 4965 spamd 9 0 31632 27M 5096 S 0.0 2.6 0:12 spamd 29212 spamd 9 0 21792 17M 5060 S 0.0 1.7 0:11 spamd 16522 spamd 9 0 20816 16M 5040 S 0.0 1.6 0:07 spamd The ps listing showing command line used and stuff:- 30559 ?S 0:02 /usr/sbin/spamd -x -u spamd -m 10 -d --pidfile=/var/run/spamassassin/spamd.pid 10873 ?S 0:23 \_ spamd child 11257 ?S 0:18 \_ spamd child 29212 ?S 0:11 \_ spamd child 4965 ?S 0:12 \_ spamd child 16522 ?S 0:07 \_ spamd child 16724 ?S 0:06 \_ spamd child 24921 ?S 0:03 \_ spamd child 25813 ?S 0:03 \_ spamd child 29211 ?S 0:01 \_ spamd child 29899 ?S 0:00 \_ spamd child As you can see, they're hardly a serious drain. Now my system is much less busy - throughput generally sits around 5-10 messages/minute, and the system serves only about 50 users. Even so, what you're seeing looks excessive to my eyes. I'm using Debian Woody, with SpamAssassin 3.0.0 from backports.org. HTH, Mike.
Re: spamd still burning CPU in 3.0.1
Although we don't load balance it would be relatively easy to do if the incoming mailserver was really having trouble. We would just duplicate the machine using another mx record. With qmail, incoming SMTP concurrency would reach our max and the first machine would stop accepting new connections forcing connections to the secondary mx record. I guess it's not really load balancing - more like failover. At 04:18 AM 10/28/2004, email builder wrote: > We have two production mailservers running SA spamd. The first handles > about 5,000 incoming emails per hour, does spam filtering with SA and virus Can I ask you how you load balance between the two machines (obviously if one handles 5000/hr and the other 2,500, it's not straight round robin)? I am not sure I understand how to fork requests between more than one machine when calling it through spamc (I don't understand how to configure this) ??? > filtering with qmailscanner and forwards the filtered mail to a server > handling the pop accounts. We're using SA 2.64 with Bayes, AWL, Razor and > about half of the RBL's. The machine is a 2.8Ghz P4 with 1.0GB RAM and SCSI > > hard drive. CPU usuage runs between 25-40% and system load runs 1.50 to > 2.20 with isolated spikes to 7.0. > > The second machine is a 2Ghz Athlon with 1.0GB RAM and an IDE drive. It > does spam and virus filtering with SA 2.64 and qmailscanner and also > handles POP3 sessions with vpopmail. We use Bayes, AWL, Razor and the same > RBL's. It handles approx 2,500 emails per hour (with peaks of 5K > emails/hour) and approx 2,000 pop3 sessions per hour (peaks of 5K > pops/hour). CPU usage runs about 20% with peaks to 50% and system load > averages 0.80 with peaks of 16.0. > > We are pretty satisfied with the above setup. We tried moving one of the > servers to SA 3.0 in order to use the new MySQL Bayes features but got > absolutely killed on CPU usage and system load - that lasted about a day > and we reverted to 2.64. Wow, OK, I am starting to think I am actually not insane and that my system really isn't b0rked and is merely at its performance limit. Thanks Rick and Jeff for the reality check! > We figure that we'd have to reduce the email load on each server by 50% in > order to use SA 3.0 and thereby need twice as many servers. However, we're > going to wait until the SA developers take the memory and load issues > seriously and fix the problem. Maybe if enough users complain they'll do > some high volume production test comparisons of 3.0 with previous versions > and sort out the problem. This is disturbing. I'm surprised the CPU thing has not been a topic of conversation (I see the memory one is) does anyone know if the developers are looking at this at all??? Thanks again! > At 09:33 PM 10/27/2004, email builder wrote: > > > email builder wrote: > > > >>email builder wrote: > > > >>How much email are you processing ? > > > > > > > > > > > > Well, just the other day we had an average of 48 msgs/min (max > 255/min) > > > get > > > > run > > > > through SA. Can't say today yet because can't run our stats tools > until > > > the > > > > busy hours are over cuz SA is hogging the CPU. ;) > > > > > > Hi, > > > > > > Your CPU is over loaded. At 48 a minute it should run just ok on a 2.8 > > > Ghz machine, much over that it's going to start having problems. On > our > > > 2.4 Ghz (not HT) processor if I process over 35 a minute I start having > > > problems with load. > > > >I have two reactions to this: > > > >1) I like the glimmer of hope and the idea that throwing hardware at the > >problem can solve it > > > >2) Throwing hardware at problems is usually avoiding fixing the *real* > >problem. According to other posters on this list, my load is not > excessive > >for a modern-day 2.xGHz machine. I will have to re-read some messages, > but I > >believe responders to my posts on the "[OT] Email Servers" thread quoted > >similar machine specs and higher load than me and said they did not have > load > >problems. I'd love to hear that I am mistaken and that it's just a matter > of > >too little hardware, but I am skeptical... > > > > > I'd recommend upgrading to a dual server or perhaps putting in a second > > > server with round robin DNS (or if you can do it, a load balancer). > > > >We've been thinking about a multiple-machine email solution and have been > >wondering about architecture. Since SA seems to be the *only* email > server > >module that causes us grief (even amavisd-new/clamav is nicer to our > >machine!!), and although it seems strange not to go with a separate file > >server or database server machine (or to otherwise split up SMTP and IMAP, > >etc), I am starting to think (as you suggest) that just adding a separate > SA > >server is going to get us the biggest performance increase. What are > >people's opinions and experience setting up separate/multiple SA servers? > >Are there any good links for reading about such setups on the wiki or > >anywhere else? >
Re: spamd still burning CPU in 3.0.1
You are correct and I apologize to the SA team. I cannot characterize the problem as a bug - SA 3.0 is just much slower and resource intensive than SA 2.64. If I understand you correctly you are just testing Bayes. Our production testing involved using SA as a whole. And I again suggest that SA 3.0 be compared against previous versions (like 2.64) in a real world production test. Maybe the answer is to publish a cheat sheet of new features in 3.0 that need to be turned off in order to achieve the throughput of 2.64. At 01:41 AM 10/28/2004, Michael Parker wrote: On Thu, Oct 28, 2004 at 01:09:57AM -0400, Jeff Koch wrote: > > We figure that we'd have to reduce the email load on each server by 50% in > order to use SA 3.0 and thereby need twice as many servers. However, we're > going to wait until the SA developers take the memory and load issues > seriously and fix the problem. Maybe if enough users complain they'll do > some high volume production test comparisons of 3.0 with previous versions > and sort out the problem. > I believe this is an entirely unfair characterization of the development team. In all cases where recent memory issues have cropped we've worked to resolve them. As for load and speed issues, I personally take these very seriously. I would guess I benchmark bayes on the average of twice a day. The benchmark pumps 300+ msgs per minute through my server, 6.5+ million SQL queries averaging around 3200 queries per second on my MySQL server. If anyone has a reproducible memory or load issue I highly encourage you to file a bug so that we can start tracking it down. Michael Best Regards, Jeff Koch, Intersessions
Re: spamd still burning CPU in 3.0.1
--- Gavin Cato <[EMAIL PROTECTED]> wrote: > > This is what I don't get. If you can handle an avg of 500/hr, which > oh, > > wait... that's per hour. Ah, OK. That's 8/min. I'm doing an avg of > 48/min > > (255/min max). But I swear someone else had a throughput higher than > that > > who was not having CPU issues. > > > > ANYONE? What kind of throughput are people able to get from SA on a > single > > processor machine? > > Hi, > > I'm averaging 89 a minute on the previously mentioned dual 1.1ghz P3 - I > know it's not single CPU but to be honest I'd have expected a single 2.8 to > be faster. Yikes. Maybe I got too excited (about just adding hardware) too soon. I agree that a 2.8 (HT no less) should be at least as fast, although 45 is my average and although it's not as easy to just get an average for peak hours, I think my daytime average is more like 75 or possibly as many as yours. 89 a minute! Wow! What else do you run on that machine? (Do you run your other email server software there or is it a dedicated SA box? Do you also run a virus scanner for example?) __ Do you Yahoo!? Read only the mail you want - Yahoo! Mail SpamGuard. http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
> On Thu, Oct 28, 2004 at 01:09:57AM -0400, Jeff Koch wrote: > > > > We figure that we'd have to reduce the email load on each server by 50% > in > > order to use SA 3.0 and thereby need twice as many servers. However, > we're > > going to wait until the SA developers take the memory and load issues > > seriously and fix the problem. Maybe if enough users complain they'll do > > some high volume production test comparisons of 3.0 with previous > versions > > and sort out the problem. > > > > I believe this is an entirely unfair characterization of the > development team. In all cases where recent memory issues have > cropped we've worked to resolve them. As for load and speed issues, I > personally take these very seriously. I would guess I benchmark bayes > on the average of twice a day. The benchmark pumps 300+ msgs per > minute through my server, 6.5+ million SQL queries averaging around > 3200 queries per second on my MySQL server. > > If anyone has a reproducible memory or load issue I highly encourage > you to file a bug so that we can start tracking it down. Well, my excessive load is reproducable every week day between 8am and 4pm. :) I am not entirely ready to pin the blame on the SA developers but I am running vanilla 3.0.1 and have done every tweak I have found here or on the wiki all to no avail. I don't really know how to file a "bug" for such a nebulous problem... __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
RE: spamd still burning CPU in 3.0.1
> While spamd is consuming high CPU, try to run "vmstat 5" to see what is
> happening.
During peak load, lots of processes waiting to run (under column r) (as many
as 15 to 20 at times, but always at least a couple); the number of blocked
processes (column b) is also uncomfortably high - can be upward of 10 or more
when things have gone to hell, but even just "regular" heavy load keeps this
at at least 2 or 3 constantly when lots of processes are waiting to run and
CPU is 0% idle... so (swapping) is never a problem (always zero), bi/bo (I'm
not as confident on what good numbers are for these) are I think never much
over 1500 (bi averages probably 700 or 800ish, bo higher than that); busy
times see as many as 3500 context switches (cs).
I'm no expert in understanding all of this, but I have been keeping my eye on
it. I'd definitely love to hear anyone's thoughts if I may be overlooking
other problems on my system.
> Also, what distribution are you runnng on and what does "perl -v" return ?
Fedora Core 2
# perl -v
This is perl, v5.8.3 built for i386-linux-thread-multi
Copyright 1987-2003, Larry Wall
Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'. If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.
Thanks!
> - RE
>
> -Original Message-
> From: email builder [mailto:[EMAIL PROTECTED]
> Sent: 23 October, 2004 7:06 AM
> To: users@spamassassin.apache.org
> Subject: spamd still burning CPU in 3.0.1
>
>
> I hurried out and installed 3.0.1, thinking one of those memory/language
> improvements mentioned in the release notes were going to be my savior...
>
> Sadly, 3.0.1's spamd has the same CPU-intensive behavior here. I am s
> at
> a loss; tried everything I've read... spent days reading... please, anyone
> have anything more?
>
> If spamd isn't I/O bound, my memory isn't swapping, I have no other
> processes
> that are out of control, I can't for the life of me figure out why this is
> happening.
>
> Again, my specs:
>
> A sample from top:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 1401 maildrop 16 0 39744 34m 6840 R 28.3 3.4 3:04.18 spamd
>
> spamd children average around 30% CPU, but even 50% not too unusual.
>
> load average is around 15 to 18 during the middle of the day
>
> And this is how I start spamd:
>
> LANG=en_US; export LANG; TMPDIR=/tmp/spamassassin; export TMPDIR spamd -d
> -q
> -x --max-children=5 -H /etc/razor -u maildrop -r /var/run/spamd/spamd.pid
>
> (also tried with -L to no avail)
>
> /tmp/spamassassin is mounted with tmpfs
>
> prefs/bayes/awl all in SQL, but bayes/awl not being used right now
>
> we also run named on the same machine
>
> if it's important, this is 3.0.1, downloaded and compiled manually (not a
> CPAN install)
>
> I have installed no custom rulesets, nothing extra beside whatever comes
> 100%
> stock. This is a Fedora Core 2 machine (2.8P-IV hyperthreaded, 1GB RAM)
>
> spamc is called from maildrop as such:
>
> if ( $SIZE < 262144 )
> {
>exception {
> xfilter "/usr/bin/spamc -u $LOGNAME"
>}
> }
>
> (also tried running inside of amavis to no avail)
>
> Any advice or even just pointers on any more reading I can do would be
> highly
> appreciated!
>
> > > What in the world is going on? Isn't it true that spamd (beside
> > > DCC)
> > does
> > > its thing w/out disk I/O? If so, what else could be chewing up so
> > > much
> > CPU?
> >
> > I don't know - The same thing happens to me a couple of times a day,
> > and I only get about 350 messages per day. Today it was at 12:25p:
> >
> > 12:25:07 4496511804 99.13 2532 9420 65088
> > 432884 86.93
> >
> > 12:25:07091 5.47 2.35 0.89 <<< LA
> >
> > When this happens, the HDD is constantly active. I'm using v2.64 with
> > network checks. The load average for the 21 hrs of this day is about
> > 0.1
__
Do you Yahoo!?
Yahoo! Mail - You care about security. So do we.
http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
On Thursday 28 October 2004 12:18 am, email builder wrote: > > We have two production mailservers running SA spamd. The first handles > > about 5,000 incoming emails per hour, does spam filtering with SA and > > virus > > Can I ask you how you load balance between the two machines (obviously if > one handles 5000/hr and the other 2,500, it's not straight round robin)? I saw nothing in his post to suggest he balanced load, or even that the two servers were serving the same domains. I just took it at face value that with 3.0.1 they couldn't keep up, but falling back to 2.64 he could carry the load. -- _ John Andersen pgpAuQCf21d6n.pgp Description: signature
Re: spamd still burning CPU in 3.0.1
> We have two production mailservers running SA spamd. The first handles > about 5,000 incoming emails per hour, does spam filtering with SA and virus Can I ask you how you load balance between the two machines (obviously if one handles 5000/hr and the other 2,500, it's not straight round robin)? I am not sure I understand how to fork requests between more than one machine when calling it through spamc (I don't understand how to configure this) ??? > filtering with qmailscanner and forwards the filtered mail to a server > handling the pop accounts. We're using SA 2.64 with Bayes, AWL, Razor and > about half of the RBL's. The machine is a 2.8Ghz P4 with 1.0GB RAM and SCSI > > hard drive. CPU usuage runs between 25-40% and system load runs 1.50 to > 2.20 with isolated spikes to 7.0. > > The second machine is a 2Ghz Athlon with 1.0GB RAM and an IDE drive. It > does spam and virus filtering with SA 2.64 and qmailscanner and also > handles POP3 sessions with vpopmail. We use Bayes, AWL, Razor and the same > RBL's. It handles approx 2,500 emails per hour (with peaks of 5K > emails/hour) and approx 2,000 pop3 sessions per hour (peaks of 5K > pops/hour). CPU usage runs about 20% with peaks to 50% and system load > averages 0.80 with peaks of 16.0. > > We are pretty satisfied with the above setup. We tried moving one of the > servers to SA 3.0 in order to use the new MySQL Bayes features but got > absolutely killed on CPU usage and system load - that lasted about a day > and we reverted to 2.64. Wow, OK, I am starting to think I am actually not insane and that my system really isn't b0rked and is merely at its performance limit. Thanks Rick and Jeff for the reality check! > We figure that we'd have to reduce the email load on each server by 50% in > order to use SA 3.0 and thereby need twice as many servers. However, we're > going to wait until the SA developers take the memory and load issues > seriously and fix the problem. Maybe if enough users complain they'll do > some high volume production test comparisons of 3.0 with previous versions > and sort out the problem. This is disturbing. I'm surprised the CPU thing has not been a topic of conversation (I see the memory one is) does anyone know if the developers are looking at this at all??? Thanks again! > At 09:33 PM 10/27/2004, email builder wrote: > > > email builder wrote: > > > >>email builder wrote: > > > >>How much email are you processing ? > > > > > > > > > > > > Well, just the other day we had an average of 48 msgs/min (max > 255/min) > > > get > > > > run > > > > through SA. Can't say today yet because can't run our stats tools > until > > > the > > > > busy hours are over cuz SA is hogging the CPU. ;) > > > > > > Hi, > > > > > > Your CPU is over loaded. At 48 a minute it should run just ok on a 2.8 > > > Ghz machine, much over that it's going to start having problems. On > our > > > 2.4 Ghz (not HT) processor if I process over 35 a minute I start having > > > problems with load. > > > >I have two reactions to this: > > > >1) I like the glimmer of hope and the idea that throwing hardware at the > >problem can solve it > > > >2) Throwing hardware at problems is usually avoiding fixing the *real* > >problem. According to other posters on this list, my load is not > excessive > >for a modern-day 2.xGHz machine. I will have to re-read some messages, > but I > >believe responders to my posts on the "[OT] Email Servers" thread quoted > >similar machine specs and higher load than me and said they did not have > load > >problems. I'd love to hear that I am mistaken and that it's just a matter > of > >too little hardware, but I am skeptical... > > > > > I'd recommend upgrading to a dual server or perhaps putting in a second > > > server with round robin DNS (or if you can do it, a load balancer). > > > >We've been thinking about a multiple-machine email solution and have been > >wondering about architecture. Since SA seems to be the *only* email > server > >module that causes us grief (even amavisd-new/clamav is nicer to our > >machine!!), and although it seems strange not to go with a separate file > >server or database server machine (or to otherwise split up SMTP and IMAP, > >etc), I am starting to think (as you suggest) that just adding a separate > SA > >server is going to get us the biggest performance increase. What are > >people's opinions and experience setting up separate/multiple SA servers? > >Are there any good links for reading about such setups on the wiki or > >anywhere else? > > > > > SA is that CPU intensive, it really is. Maybe try adding RBL's in > front > > > of the MTA to reduce the number of messages you have to scan, that's > > > what we do. > > > >Ha! Yeah, this message rate is *WITH* something like 10 RBL's in Postfix > up > >front. W/out that, we'd *really* be drowning. :) > > > >Many thanks! __ Do You Yahoo!? Tired of spam? Yahoo
Re: spamd still burning CPU in 3.0.1
> This is what I don't get. If you can handle an avg of 500/hr, which oh, > wait... that's per hour. Ah, OK. That's 8/min. I'm doing an avg of 48/min > (255/min max). But I swear someone else had a throughput higher than that > who was not having CPU issues. > > ANYONE? What kind of throughput are people able to get from SA on a single > processor machine? Hi, I'm averaging 89 a minute on the previously mentioned dual 1.1ghz P3 - I know it's not single CPU but to be honest I'd have expected a single 2.8 to be faster.
Re: spamd still burning CPU in 3.0.1
On Thu, Oct 28, 2004 at 01:09:57AM -0400, Jeff Koch wrote: > > We figure that we'd have to reduce the email load on each server by 50% in > order to use SA 3.0 and thereby need twice as many servers. However, we're > going to wait until the SA developers take the memory and load issues > seriously and fix the problem. Maybe if enough users complain they'll do > some high volume production test comparisons of 3.0 with previous versions > and sort out the problem. > I believe this is an entirely unfair characterization of the development team. In all cases where recent memory issues have cropped we've worked to resolve them. As for load and speed issues, I personally take these very seriously. I would guess I benchmark bayes on the average of twice a day. The benchmark pumps 300+ msgs per minute through my server, 6.5+ million SQL queries averaging around 3200 queries per second on my MySQL server. If anyone has a reproducible memory or load issue I highly encourage you to file a bug so that we can start tracking it down. Michael pgpAcKjoh3onZ.pgp Description: PGP signature
Re: spamd still burning CPU in 3.0.1
We have two production mailservers running SA spamd. The first handles about 5,000 incoming emails per hour, does spam filtering with SA and virus filtering with qmailscanner and forwards the filtered mail to a server handling the pop accounts. We're using SA 2.64 with Bayes, AWL, Razor and about half of the RBL's. The machine is a 2.8Ghz P4 with 1.0GB RAM and SCSI hard drive. CPU usuage runs between 25-40% and system load runs 1.50 to 2.20 with isolated spikes to 7.0. The second machine is a 2Ghz Athlon with 1.0GB RAM and an IDE drive. It does spam and virus filtering with SA 2.64 and qmailscanner and also handles POP3 sessions with vpopmail. We use Bayes, AWL, Razor and the same RBL's. It handles approx 2,500 emails per hour (with peaks of 5K emails/hour) and approx 2,000 pop3 sessions per hour (peaks of 5K pops/hour). CPU usage runs about 20% with peaks to 50% and system load averages 0.80 with peaks of 16.0. We are pretty satisfied with the above setup. We tried moving one of the servers to SA 3.0 in order to use the new MySQL Bayes features but got absolutely killed on CPU usage and system load - that lasted about a day and we reverted to 2.64. We figure that we'd have to reduce the email load on each server by 50% in order to use SA 3.0 and thereby need twice as many servers. However, we're going to wait until the SA developers take the memory and load issues seriously and fix the problem. Maybe if enough users complain they'll do some high volume production test comparisons of 3.0 with previous versions and sort out the problem. At 09:33 PM 10/27/2004, email builder wrote: > email builder wrote: > >>email builder wrote: > >>How much email are you processing ? > > > > > > Well, just the other day we had an average of 48 msgs/min (max 255/min) > get > > run > > through SA. Can't say today yet because can't run our stats tools until > the > > busy hours are over cuz SA is hogging the CPU. ;) > > Hi, > > Your CPU is over loaded. At 48 a minute it should run just ok on a 2.8 > Ghz machine, much over that it's going to start having problems. On our > 2.4 Ghz (not HT) processor if I process over 35 a minute I start having > problems with load. I have two reactions to this: 1) I like the glimmer of hope and the idea that throwing hardware at the problem can solve it 2) Throwing hardware at problems is usually avoiding fixing the *real* problem. According to other posters on this list, my load is not excessive for a modern-day 2.xGHz machine. I will have to re-read some messages, but I believe responders to my posts on the "[OT] Email Servers" thread quoted similar machine specs and higher load than me and said they did not have load problems. I'd love to hear that I am mistaken and that it's just a matter of too little hardware, but I am skeptical... > I'd recommend upgrading to a dual server or perhaps putting in a second > server with round robin DNS (or if you can do it, a load balancer). We've been thinking about a multiple-machine email solution and have been wondering about architecture. Since SA seems to be the *only* email server module that causes us grief (even amavisd-new/clamav is nicer to our machine!!), and although it seems strange not to go with a separate file server or database server machine (or to otherwise split up SMTP and IMAP, etc), I am starting to think (as you suggest) that just adding a separate SA server is going to get us the biggest performance increase. What are people's opinions and experience setting up separate/multiple SA servers? Are there any good links for reading about such setups on the wiki or anywhere else? > SA is that CPU intensive, it really is. Maybe try adding RBL's in front > of the MTA to reduce the number of messages you have to scan, that's > what we do. Ha! Yeah, this message rate is *WITH* something like 10 RBL's in Postfix up front. W/out that, we'd *really* be drowning. :) Many thanks! __ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail Best Regards, Jeff Koch
Re: spamd still burning CPU in 3.0.1
> >>I'd recommend upgrading to a dual server or perhaps putting in a second > >>server with round robin DNS (or if you can do it, a load balancer). > > > > > > also, what do people think about a multiple cpu machine vs more than one > > machine? dumb question? (two machines always are faster than one > dual-cpu machine) > > I believe I answered your other post with my machine specs, can't remember. Yes you did, and your replies are highly appreciated!! > Anyways, I highly recommend dual cpu with SA. I also highly recommend > running spamd on another server and letting your mailserver/toaster call > it with spamc. We run SA on our NFS server which is a Sparc Enterpise, > 2gb ram, dual cpu, and it is handling all the spam processing for three > toasters with no problem. It also serves our Maildirs to each toaster. Thanks. We were thinking about a NFS server, but SA concerns seemed more important. If both can coexist peacefully, this may be the exact same solution that we use. > Even with all the NFS traffic load is under 5 on a very busy afternoon. > My max this month was 3200 messages per hour according to qmail-mrtg. My > avg is around 500 messages per hour. This is what I don't get. If you can handle an avg of 500/hr, which oh, wait... that's per hour. Ah, OK. That's 8/min. I'm doing an avg of 48/min (255/min max). But I swear someone else had a throughput higher than that who was not having CPU issues. ANYONE? What kind of throughput are people able to get from SA on a single processor machine? Does anyone have a gathering of that kind of info? (would be nice to see on the wiki) TIA! __ Do you Yahoo!? Read only the mail you want - Yahoo! Mail SpamGuard. http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
email builder wrote: I'd recommend upgrading to a dual server or perhaps putting in a second server with round robin DNS (or if you can do it, a load balancer). also, what do people think about a multiple cpu machine vs more than one machine? dumb question? (two machines always are faster than one dual-cpu machine) I believe I answered your other post with my machine specs, can't remember. Anyways, I highly recommend dual cpu with SA. I also highly recommend running spamd on another server and letting your mailserver/toaster call it with spamc. We run SA on our NFS server which is a Sparc Enterpise, 2gb ram, dual cpu, and it is handling all the spam processing for three toasters with no problem. It also serves our Maildirs to each toaster. Even with all the NFS traffic load is under 5 on a very busy afternoon. My max this month was 3200 messages per hour according to qmail-mrtg. My avg is around 500 messages per hour. DAve -- Systems Administrator http://www.tls.net Get rid of Unwanted Emails...get TLS Spam Blocker!
Re: spamd still burning CPU in 3.0.1
> I'd recommend upgrading to a dual server or perhaps putting in a second > server with round robin DNS (or if you can do it, a load balancer). also, what do people think about a multiple cpu machine vs more than one machine? dumb question? (two machines always are faster than one dual-cpu machine) __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Re: spamd still burning CPU in 3.0.1
> email builder wrote: > >>email builder wrote: > >>How much email are you processing ? > > > > > > Well, just the other day we had an average of 48 msgs/min (max 255/min) > get > > run > > through SA. Can't say today yet because can't run our stats tools until > the > > busy hours are over cuz SA is hogging the CPU. ;) > > Hi, > > Your CPU is over loaded. At 48 a minute it should run just ok on a 2.8 > Ghz machine, much over that it's going to start having problems. On our > 2.4 Ghz (not HT) processor if I process over 35 a minute I start having > problems with load. I have two reactions to this: 1) I like the glimmer of hope and the idea that throwing hardware at the problem can solve it 2) Throwing hardware at problems is usually avoiding fixing the *real* problem. According to other posters on this list, my load is not excessive for a modern-day 2.xGHz machine. I will have to re-read some messages, but I believe responders to my posts on the "[OT] Email Servers" thread quoted similar machine specs and higher load than me and said they did not have load problems. I'd love to hear that I am mistaken and that it's just a matter of too little hardware, but I am skeptical... > I'd recommend upgrading to a dual server or perhaps putting in a second > server with round robin DNS (or if you can do it, a load balancer). We've been thinking about a multiple-machine email solution and have been wondering about architecture. Since SA seems to be the *only* email server module that causes us grief (even amavisd-new/clamav is nicer to our machine!!), and although it seems strange not to go with a separate file server or database server machine (or to otherwise split up SMTP and IMAP, etc), I am starting to think (as you suggest) that just adding a separate SA server is going to get us the biggest performance increase. What are people's opinions and experience setting up separate/multiple SA servers? Are there any good links for reading about such setups on the wiki or anywhere else? > SA is that CPU intensive, it really is. Maybe try adding RBL's in front > of the MTA to reduce the number of messages you have to scan, that's > what we do. Ha! Yeah, this message rate is *WITH* something like 10 RBL's in Postfix up front. W/out that, we'd *really* be drowning. :) Many thanks! __ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
email builder wrote: email builder wrote: How much email are you processing ? Well, just the other day we had an average of 48 msgs/min (max 255/min) get run through SA. Can't say today yet because can't run our stats tools until the busy hours are over cuz SA is hogging the CPU. ;) Hi, Your CPU is over loaded. At 48 a minute it should run just ok on a 2.8 Ghz machine, much over that it's going to start having problems. On our 2.4 Ghz (not HT) processor if I process over 35 a minute I start having problems with load. I'd recommend upgrading to a dual server or perhaps putting in a second server with round robin DNS (or if you can do it, a load balancer). SA is that CPU intensive, it really is. Maybe try adding RBL's in front of the MTA to reduce the number of messages you have to scan, that's what we do. Regards, Rick
Re: spamd still burning CPU in 3.0.1
> email builder wrote: > >>BTW, SpamAssassin *is* CPU-intensive. It's designed that way ;) > > > > > > But not as CPU intensive as I am seeing. According to others on this > list, I > > should not be seeing a mere five spamd children completely dominating a > > 2.8GHz(HT) processor. > > How much email are you processing ? Well, just the other day we had an average of 48 msgs/min (max 255/min) get run through SA. Can't say today yet because can't run our stats tools until the busy hours are over cuz SA is hogging the CPU. ;) > Have you tried turning off razor ? Yes. DCC too. > Are you running a big local dns cache ? Nothing unusual. From what I understand, DNS queries should not have any relationship to CPU usage. > Is the machine swapping at all ? Not a bit. We definitely have some process blockage, but I presume that is because the CPU is being overused. Thanks! __ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
Definitely not, I'm using SA 3.0.1 on a dual 1.13ghz P3 with 2gb RAM with SCSI, processing a fair bit of mail. I have 25 spamd children running, and the load is typically like this ; > w 9:46AM up 9 days, 13:06, 1 user, load averages: 1.14, 1.46, 1.59 Cheers Gav On 28/10/04 9:13 AM, "email builder" <[EMAIL PROTECTED]> wrote: >> BTW, SpamAssassin *is* CPU-intensive. It's designed that way ;) > > But not as CPU intensive as I am seeing. According to others on this list, I > should not be seeing a mere five spamd children completely dominating a > 2.8GHz(HT) processor. >
Re: spamd still burning CPU in 3.0.1
email builder wrote: BTW, SpamAssassin *is* CPU-intensive. It's designed that way ;) But not as CPU intensive as I am seeing. According to others on this list, I should not be seeing a mere five spamd children completely dominating a 2.8GHz(HT) processor. How much email are you processing ? Have you tried turning off razor ? Are you running a big local dns cache ? Is the machine swapping at all ? Regards, Rick
Re: spamd still burning CPU in 3.0.1
> Does spamd burn up the CPU if you do not have the Bayes turned on? > If not, then I humbly suggest to turn off the Bayes in SA and > use bogofilter to handle the Bayes processing. Unfortunately, with use_bayes set to zero, spamd children average probably around 20% cpu and bounce regularly into the 30's and higher. :( __ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
> BTW, SpamAssassin *is* CPU-intensive. It's designed that way ;)
But not as CPU intensive as I am seeing. According to others on this list, I
should not be seeing a mere five spamd children completely dominating a
2.8GHz(HT) processor.
> Tim B writes:
> > email builder wrote:
> > > I hurried out and installed 3.0.1, thinking one of those
> memory/language
> > > improvements mentioned in the release notes were going to be my
> savior...
> > >
> > > Sadly, 3.0.1's spamd has the same CPU-intensive behavior here. I am
> s at
> > > a loss; tried everything I've read... spent days reading... please,
> anyone
> > > have anything more?
> > >
> > > If spamd isn't I/O bound, my memory isn't swapping, I have no other
> processes
> > > that are out of control, I can't for the life of me figure out why this
> is
> > > happening.
> > >
> > > Again, my specs:
> > >
> > > A sample from top:
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> > > 1401 maildrop 16 0 39744 34m 6840 R 28.3 3.4 3:04.18 spamd
> > >
> > > spamd children average around 30% CPU, but even 50% not too unusual.
> > >
> > > load average is around 15 to 18 during the middle of the day
> > >
> > > And this is how I start spamd:
> > >
> > > LANG=en_US; export LANG; TMPDIR=/tmp/spamassassin; export TMPDIR
> > > spamd -d -q -x --max-children=5 -H /etc/razor -u maildrop -r
> > > /var/run/spamd/spamd.pid
> > >
> > > (also tried with -L to no avail)
> > >
> > > /tmp/spamassassin is mounted with tmpfs
> > >
> > > prefs/bayes/awl all in SQL, but bayes/awl not being used right now
> > >
> > > we also run named on the same machine
> > >
> > > if it's important, this is 3.0.1, downloaded and compiled manually (not
> a
> > > CPAN install)
> > >
> > > I have installed no custom rulesets, nothing extra beside whatever
> comes 100%
> > > stock. This is a Fedora Core 2 machine (2.8P-IV hyperthreaded, 1GB
> RAM)
> > >
> > > spamc is called from maildrop as such:
> > >
> > > if ( $SIZE < 262144 )
> > > {
> > >exception {
> > > xfilter "/usr/bin/spamc -u $LOGNAME"
> > >}
> > > }
> > >
> > > (also tried running inside of amavis to no avail)
> > >
> > > Any advice or even just pointers on any more reading I can do would be
> highly
> > > appreciated!
> > >
> > >
> > >>>What in the world is going on? Isn't it true that spamd (beside DCC)
> > >>
> > >>does
> > >>
> > >>>its thing w/out disk I/O? If so, what else could be chewing up so
> much
> > >>
> > >>CPU?
> > >>
> > >>I don't know - The same thing happens to me a couple of times a day,
> and I
> > >>only get about 350 messages per day. Today it was at 12:25p:
> > >>
> > >>12:25:07 4496511804 99.13 2532 9420 65088
> > >>432884 86.93
> > >>
> > >>12:25:07091 5.47 2.35 0.89 <<<
> LA
> > >>
> > >>When this happens, the HDD is constantly active. I'm using v2.64 with
> > >>network checks. The load average for the 21 hrs of this day is about
> 0.1
> > >
> > >
> > >
> > >
> > >
> > > __
> > > Do you Yahoo!?
> > > Yahoo! Mail - Helps protect you from nasty viruses.
> > > http://promotions.yahoo.com/new_mail
> > >
> >
> > Have you tried doing a force-expire on your bayes db?
> >
> > I found this helped me. Disabling autoexpire, and twice a day running
> > sa-learn --force-expire
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.2.4 (GNU/Linux)
> Comment: Exmh CVS
>
> iD8DBQFBerAlMJF5cimLx9ARAkSzAJ4ziHgz6iLos/0Obf7OxcFEBxs3gwCfRCvG
> 3pCNDv79pho0WFpZWnvVDEA=
> =6UpA
> -END PGP SIGNATURE-
>
>
__
Do you Yahoo!?
Yahoo! Mail Address AutoComplete - You start. We finish.
http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
--- Tim B <[EMAIL PROTECTED]> wrote: > email builder wrote: > > > I hurried out and installed 3.0.1, thinking one of those memory/language > > improvements mentioned in the release notes were going to be my savior... > > > > Sadly, 3.0.1's spamd has the same CPU-intensive behavior here. I am > s at > > a loss; tried everything I've read... spent days reading... please, > anyone > > have anything more? > > > > If spamd isn't I/O bound, my memory isn't swapping, I have no other > processes > > that are out of control, I can't for the life of me figure out why this > is > > happening. > > > > Again, my specs: > > > I don't remember who on the list posted this origonally > but try adding --max-conn-per-child=1 > > spamd -d -q -x --max-children=5 --max-conn-per-child=1 -H /etc/razor -u > maildrop -r This appears to *maybe* help an almost imperceptible amount. I'd using it for fun, but the CPU still is being used much more than it should. Our load is way too high and as a result, everything slows down and falls behind during our busy hours. Thanks for the advice anyone else have *any* idea where else I can look? __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Re: spamd still burning CPU in 3.0.1
--- Tim B <[EMAIL PROTECTED]> wrote: > email builder wrote: > > I hurried out and installed 3.0.1, thinking one of those memory/language > > improvements mentioned in the release notes were going to be my savior... > > > > Sadly, 3.0.1's spamd has the same CPU-intensive behavior here. I am > s at > > a loss; tried everything I've read... spent days reading... please, > anyone > > have anything more? > > > > If spamd isn't I/O bound, my memory isn't swapping, I have no other > processes > > that are out of control, I can't for the life of me figure out why this > is > > happening. > > > Have you tried doing a force-expire on your bayes db? > > I found this helped me. Disabling autoexpire, and twice a day running > sa-learn --force-expire I have use_bayes set to zero until I figure out why the CPU is getting hammered. If I turn bayes on, things become too slow to keep up with incoming mail. Thanks anyway. __ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail
Re: spamd still burning CPU in 3.0.1
Does spamd burn up the CPU if you do not have the Bayes turned on? If not, then I humbly suggest to turn off the Bayes in SA and use bogofilter to handle the Bayes processing.
Re: spamd still burning CPU in 3.0.1
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
BTW, SpamAssassin *is* CPU-intensive. It's designed that way ;)
- --j.
Tim B writes:
> email builder wrote:
> > I hurried out and installed 3.0.1, thinking one of those memory/language
> > improvements mentioned in the release notes were going to be my savior...
> >
> > Sadly, 3.0.1's spamd has the same CPU-intensive behavior here. I am s
> > at
> > a loss; tried everything I've read... spent days reading... please, anyone
> > have anything more?
> >
> > If spamd isn't I/O bound, my memory isn't swapping, I have no other
> > processes
> > that are out of control, I can't for the life of me figure out why this is
> > happening.
> >
> > Again, my specs:
> >
> > A sample from top:
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> > 1401 maildrop 16 0 39744 34m 6840 R 28.3 3.4 3:04.18 spamd
> >
> > spamd children average around 30% CPU, but even 50% not too unusual.
> >
> > load average is around 15 to 18 during the middle of the day
> >
> > And this is how I start spamd:
> >
> > LANG=en_US; export LANG; TMPDIR=/tmp/spamassassin; export TMPDIR
> > spamd -d -q -x --max-children=5 -H /etc/razor -u maildrop -r
> > /var/run/spamd/spamd.pid
> >
> > (also tried with -L to no avail)
> >
> > /tmp/spamassassin is mounted with tmpfs
> >
> > prefs/bayes/awl all in SQL, but bayes/awl not being used right now
> >
> > we also run named on the same machine
> >
> > if it's important, this is 3.0.1, downloaded and compiled manually (not a
> > CPAN install)
> >
> > I have installed no custom rulesets, nothing extra beside whatever comes
> > 100%
> > stock. This is a Fedora Core 2 machine (2.8P-IV hyperthreaded, 1GB RAM)
> >
> > spamc is called from maildrop as such:
> >
> > if ( $SIZE < 262144 )
> > {
> >exception {
> > xfilter "/usr/bin/spamc -u $LOGNAME"
> >}
> > }
> >
> > (also tried running inside of amavis to no avail)
> >
> > Any advice or even just pointers on any more reading I can do would be
> > highly
> > appreciated!
> >
> >
> >>>What in the world is going on? Isn't it true that spamd (beside DCC)
> >>
> >>does
> >>
> >>>its thing w/out disk I/O? If so, what else could be chewing up so much
> >>
> >>CPU?
> >>
> >>I don't know - The same thing happens to me a couple of times a day, and I
> >>only get about 350 messages per day. Today it was at 12:25p:
> >>
> >>12:25:07 4496511804 99.13 2532 9420 65088
> >>432884 86.93
> >>
> >>12:25:07091 5.47 2.35 0.89 <<< LA
> >>
> >>When this happens, the HDD is constantly active. I'm using v2.64 with
> >>network checks. The load average for the 21 hrs of this day is about 0.1
> >
> >
> >
> >
> >
> > __
> > Do you Yahoo!?
> > Yahoo! Mail - Helps protect you from nasty viruses.
> > http://promotions.yahoo.com/new_mail
> >
>
> Have you tried doing a force-expire on your bayes db?
>
> I found this helped me. Disabling autoexpire, and twice a day running
> sa-learn --force-expire
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS
iD8DBQFBerAlMJF5cimLx9ARAkSzAJ4ziHgz6iLos/0Obf7OxcFEBxs3gwCfRCvG
3pCNDv79pho0WFpZWnvVDEA=
=6UpA
-END PGP SIGNATURE-
Re: spamd still burning CPU in 3.0.1
email builder wrote:
I hurried out and installed 3.0.1, thinking one of those memory/language
improvements mentioned in the release notes were going to be my savior...
Sadly, 3.0.1's spamd has the same CPU-intensive behavior here. I am s at
a loss; tried everything I've read... spent days reading... please, anyone
have anything more?
If spamd isn't I/O bound, my memory isn't swapping, I have no other processes
that are out of control, I can't for the life of me figure out why this is
happening.
Again, my specs:
A sample from top:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
1401 maildrop 16 0 39744 34m 6840 R 28.3 3.4 3:04.18 spamd
spamd children average around 30% CPU, but even 50% not too unusual.
load average is around 15 to 18 during the middle of the day
And this is how I start spamd:
LANG=en_US; export LANG; TMPDIR=/tmp/spamassassin; export TMPDIR
spamd -d -q -x --max-children=5 -H /etc/razor -u maildrop -r
/var/run/spamd/spamd.pid
(also tried with -L to no avail)
/tmp/spamassassin is mounted with tmpfs
prefs/bayes/awl all in SQL, but bayes/awl not being used right now
we also run named on the same machine
if it's important, this is 3.0.1, downloaded and compiled manually (not a
CPAN install)
I have installed no custom rulesets, nothing extra beside whatever comes 100%
stock. This is a Fedora Core 2 machine (2.8P-IV hyperthreaded, 1GB RAM)
spamc is called from maildrop as such:
if ( $SIZE < 262144 )
{
exception {
xfilter "/usr/bin/spamc -u $LOGNAME"
}
}
(also tried running inside of amavis to no avail)
Any advice or even just pointers on any more reading I can do would be highly
appreciated!
What in the world is going on? Isn't it true that spamd (beside DCC)
does
its thing w/out disk I/O? If so, what else could be chewing up so much
CPU?
I don't know - The same thing happens to me a couple of times a day, and I
only get about 350 messages per day. Today it was at 12:25p:
12:25:07 4496511804 99.13 2532 9420 65088
432884 86.93
12:25:07091 5.47 2.35 0.89 <<< LA
When this happens, the HDD is constantly active. I'm using v2.64 with
network checks. The load average for the 21 hrs of this day is about 0.1
__
Do you Yahoo!?
Yahoo! Mail - Helps protect you from nasty viruses.
http://promotions.yahoo.com/new_mail
Have you tried doing a force-expire on your bayes db?
I found this helped me. Disabling autoexpire, and twice a day running
sa-learn --force-expire
