Re: Do I have too many rules? [WAS: timeout help]
From: "Josh Trutwin" <[EMAIL PROTECTED]>
On Sat, 19 Aug 2006 18:06:59 -0400
"Daryl C. W. O'Shea" <[EMAIL PROTECTED]> wrote:
Josh Trutwin wrote:
> On Fri, 18 Aug 2006 12:16:51 -0400
> "Daryl C. W. O'Shea" <[EMAIL PROTECTED]> wrote:
>
>> Josh Trutwin wrote:
>>> I've recently had a server experience some really slow spam
>>> processing - I'm not sure what's going on but I notice a lot
>>> of timeouts in the mail log:
>>>
>>> Aug 18 09:20:21 www spamd[27673]: timeout with empty $@
>>> at /usr/local/share/perl/5.8.4/Mail/SpamAssassin/Timeout.pm
>>> line 182, line 1126. Aug 18 09:22:02 www spamd[27674]:
>>> timeout with empty $@
>>> Any suggestions?
>>>
>>> Debian linux - spamd 3.0.4 with pyzor/dcc/razor
>>>
>>> spamd running with:
>>>
>>> /usr/bin/spamd -d -D -q -x -H /etc/razor --max-children=12
>>> --socketpath=/var/spool/spamassassin/spamd.sock -u spamd
>> Unless you've got at least 600 or more MB of free RAM just for
>> spamd's use, you've got too many children and are swap
>> thrashing. Back of the --max-children number.
>
> I was getting the same results with less values - the box has 1
> GB
> - more is on the way though. I disabled network tests with -L
> and things work great again so something along that line is the
> culprit.
Using -L processes messages faster, thus requiring less children
to handle the load. This error is caused by a system not being
able to restore the child's config fast enough after it's done
processing a message. It's always due to one of two things...
high load, or high load caused by swap thrashing.
If you're using a lot of add-on rulesets your children may be
taking up even more memory than budgeted above. See what they're
using and confirm that you're not going to hit swap. If the
problem still persists I'd like to hear about it.
Still having problems - even with -L. Server has 1 GB of memory,
more is on the way I hope. Anyway - I have the following rules:
fastconcepts:/etc/mail/spamassassin# ls
10_misc.cf 88_FVGT_headers.cf
70_sare_adult.cf 88_FVGT_rawbody.cf
70_sare_bayes_poison_nxm.cf88_FVGT_subject.cf
70_sare_evilnum0.cf88_FVGT_uri.cf
70_sare_genlsubj.cf99_FVGT_DomainDigits.cf
70_sare_header.cf 99_FVGT_meta.cf
70_sare_highrisk.cf99_sare_fraud_post25x.cf
70_sare_html.cfRulesDuJour
70_sare_obfu.cfantidrug.cf
70_sare_oem.cf backhair.cf
70_sare_random.cf bogus-virus-warnings.cf
70_sare_ratware.cf chickenpox.cf
70_sare_specific.cfcoaching.cf
70_sare_spoof.cf init.pre
70_sare_stocks.cf local.cf
70_sare_unsub.cf mr_wiggly.cf
70_sare_uri.cf random.cf
70_sare_uri0.cftripwire.cf
70_sare_whitelist_rcvd.cf tsa-list.cf
70_sare_whitelist_spf.cf useless.cf
72_sare_bml_post25x.cf v310.pre
72_sare_redirect_post3.0.0.cf v312.pre
88_FVGT_body.cfweeds.cf
I created coaching.cf and tsa-list.cf - they are basically one
(well two technically) liners.
If I need to cut back I'm not sure where to start - the SARE rules
are the ones listed on rulesemporium.com - the FVGT are on
http://www.exit0.us/index.php?pagename=RulesDuJourRuleSets. Of the
rest, only bogus-virus-warnings.cf is of any substantial size (and
still 1/3 the size of the largest SARE rule).
Should v310.pre be removed if v312.pre is also there?
No. Cut down on the number of children. 12 is ridiculous.
{^_^}
Re: Do I have too many rules? [WAS: timeout help]
On Thu, 24 Aug 2006 10:11:28 -0400 Bowie Bailey <[EMAIL PROTECTED]> wrote: > Josh Trutwin wrote: > > > > Still having problems - even with -L. Server has 1 GB of > > memory, more is on the way I hope. > > You said previously that you were running 12 children. With 1GB > of RAM, I would suggest that you drop it to 6 and see what > happens. Thanks - I had backed off to 8, still had probs so now I'm at 4. > > Anyway - I have the following rules: > > > > fastconcepts:/etc/mail/spamassassin# ls > > 10_misc.cf 88_FVGT_headers.cf > > 70_sare_adult.cf 88_FVGT_rawbody.cf > > 70_sare_bayes_poison_nxm.cf88_FVGT_subject.cf > > 70_sare_evilnum0.cf88_FVGT_uri.cf > > 70_sare_genlsubj.cf99_FVGT_DomainDigits.cf > > 70_sare_header.cf 99_FVGT_meta.cf > > 70_sare_highrisk.cf99_sare_fraud_post25x.cf > > 70_sare_html.cfRulesDuJour > > 70_sare_obfu.cfantidrug.cf > > 70_sare_oem.cf backhair.cf > > 70_sare_random.cf bogus-virus-warnings.cf > > 70_sare_ratware.cf chickenpox.cf > > 70_sare_specific.cfcoaching.cf > > 70_sare_spoof.cf init.pre > > 70_sare_stocks.cf local.cf > > 70_sare_unsub.cf mr_wiggly.cf > > 70_sare_uri.cf random.cf > > 70_sare_uri0.cftripwire.cf > > 70_sare_whitelist_rcvd.cf tsa-list.cf > > 70_sare_whitelist_spf.cf useless.cf > > 72_sare_bml_post25x.cf v310.pre > > 72_sare_redirect_post3.0.0.cf v312.pre > > 88_FVGT_body.cfweeds.cf > > > > I created coaching.cf and tsa-list.cf - they are basically one > > (well two technically) liners. > > > > If I need to cut back I'm not sure where to start - the SARE > > rules are the ones listed on rulesemporium.com - the FVGT are on > > http://www.exit0.us/index.php?pagename=RulesDuJourRuleSets. Of > > the rest, only bogus-virus-warnings.cf is of any substantial > > size (and still 1/3 the size of the largest SARE rule). > > antidrug.cf can be removed. These rules are already included in > the latest SA versions. Thanks - done. This should probably be removed from the rules_du_jour script then or marked Deprecated. I also was able to remove mr_wrigley.cf. > 70_sare_uri.cf and 70_sare_uri0.cf are overlapping. You should > use one or the other, not both. 70_sare_uri.cf contains all of > the SARE uri rules. 70_sare_uri0.cf contains only the uri rules > that did not hit any ham during testing. That's strange - 70_sare_uri.cf is not even available on the rules emporium page anymore - must've been downloaded before it was split. I removed em both and went with 70_sare_uri_eng.cf > > Should v310.pre be removed if v312.pre is also there? > > No, v312.pre contains plugins that were added with SA 3.1.2 and > v310.pre contains plugins added with SA 3.1.0. You should not > remove any of these files. Thanks - that was my guess. :) Josh
RE: Do I have too many rules? [WAS: timeout help]
Josh Trutwin wrote: > > Still having problems - even with -L. Server has 1 GB of memory, > more is on the way I hope. You said previously that you were running 12 children. With 1GB of RAM, I would suggest that you drop it to 6 and see what happens. > Anyway - I have the following rules: > > fastconcepts:/etc/mail/spamassassin# ls > 10_misc.cf 88_FVGT_headers.cf > 70_sare_adult.cf 88_FVGT_rawbody.cf > 70_sare_bayes_poison_nxm.cf88_FVGT_subject.cf > 70_sare_evilnum0.cf88_FVGT_uri.cf > 70_sare_genlsubj.cf99_FVGT_DomainDigits.cf > 70_sare_header.cf 99_FVGT_meta.cf > 70_sare_highrisk.cf99_sare_fraud_post25x.cf > 70_sare_html.cfRulesDuJour > 70_sare_obfu.cfantidrug.cf > 70_sare_oem.cf backhair.cf > 70_sare_random.cf bogus-virus-warnings.cf > 70_sare_ratware.cf chickenpox.cf > 70_sare_specific.cfcoaching.cf > 70_sare_spoof.cf init.pre > 70_sare_stocks.cf local.cf > 70_sare_unsub.cf mr_wiggly.cf > 70_sare_uri.cf random.cf > 70_sare_uri0.cftripwire.cf > 70_sare_whitelist_rcvd.cf tsa-list.cf > 70_sare_whitelist_spf.cf useless.cf > 72_sare_bml_post25x.cf v310.pre > 72_sare_redirect_post3.0.0.cf v312.pre > 88_FVGT_body.cfweeds.cf > > I created coaching.cf and tsa-list.cf - they are basically one > (well two technically) liners. > > If I need to cut back I'm not sure where to start - the SARE rules > are the ones listed on rulesemporium.com - the FVGT are on > http://www.exit0.us/index.php?pagename=RulesDuJourRuleSets. Of the > rest, only bogus-virus-warnings.cf is of any substantial size (and > still 1/3 the size of the largest SARE rule). antidrug.cf can be removed. These rules are already included in the latest SA versions. 70_sare_uri.cf and 70_sare_uri0.cf are overlapping. You should use one or the other, not both. 70_sare_uri.cf contains all of the SARE uri rules. 70_sare_uri0.cf contains only the uri rules that did not hit any ham during testing. > Should v310.pre be removed if v312.pre is also there? No, v312.pre contains plugins that were added with SA 3.1.2 and v310.pre contains plugins added with SA 3.1.0. You should not remove any of these files. -- Bowie
Re: Do I have too many rules? [WAS: timeout help]
On Sat, 19 Aug 2006 18:06:59 -0400 "Daryl C. W. O'Shea" <[EMAIL PROTECTED]> wrote: > Josh Trutwin wrote: > > On Fri, 18 Aug 2006 12:16:51 -0400 > > "Daryl C. W. O'Shea" <[EMAIL PROTECTED]> wrote: > > > >> Josh Trutwin wrote: > >>> I've recently had a server experience some really slow spam > >>> processing - I'm not sure what's going on but I notice a lot > >>> of timeouts in the mail log: > >>> > >>> Aug 18 09:20:21 www spamd[27673]: timeout with empty $@ > >>> at /usr/local/share/perl/5.8.4/Mail/SpamAssassin/Timeout.pm > >>> line 182, line 1126. Aug 18 09:22:02 www spamd[27674]: > >>> timeout with empty $@ > >>> Any suggestions? > >>> > >>> Debian linux - spamd 3.0.4 with pyzor/dcc/razor > >>> > >>> spamd running with: > >>> > >>> /usr/bin/spamd -d -D -q -x -H /etc/razor --max-children=12 > >>> --socketpath=/var/spool/spamassassin/spamd.sock -u spamd > >> Unless you've got at least 600 or more MB of free RAM just for > >> spamd's use, you've got too many children and are swap > >> thrashing. Back of the --max-children number. > > > > I was getting the same results with less values - the box has 1 > > GB > > - more is on the way though. I disabled network tests with -L > > and things work great again so something along that line is the > > culprit. > > Using -L processes messages faster, thus requiring less children > to handle the load. This error is caused by a system not being > able to restore the child's config fast enough after it's done > processing a message. It's always due to one of two things... > high load, or high load caused by swap thrashing. > > If you're using a lot of add-on rulesets your children may be > taking up even more memory than budgeted above. See what they're > using and confirm that you're not going to hit swap. If the > problem still persists I'd like to hear about it. Still having problems - even with -L. Server has 1 GB of memory, more is on the way I hope. Anyway - I have the following rules: fastconcepts:/etc/mail/spamassassin# ls 10_misc.cf 88_FVGT_headers.cf 70_sare_adult.cf 88_FVGT_rawbody.cf 70_sare_bayes_poison_nxm.cf88_FVGT_subject.cf 70_sare_evilnum0.cf88_FVGT_uri.cf 70_sare_genlsubj.cf99_FVGT_DomainDigits.cf 70_sare_header.cf 99_FVGT_meta.cf 70_sare_highrisk.cf99_sare_fraud_post25x.cf 70_sare_html.cfRulesDuJour 70_sare_obfu.cfantidrug.cf 70_sare_oem.cf backhair.cf 70_sare_random.cf bogus-virus-warnings.cf 70_sare_ratware.cf chickenpox.cf 70_sare_specific.cfcoaching.cf 70_sare_spoof.cf init.pre 70_sare_stocks.cf local.cf 70_sare_unsub.cf mr_wiggly.cf 70_sare_uri.cf random.cf 70_sare_uri0.cftripwire.cf 70_sare_whitelist_rcvd.cf tsa-list.cf 70_sare_whitelist_spf.cf useless.cf 72_sare_bml_post25x.cf v310.pre 72_sare_redirect_post3.0.0.cf v312.pre 88_FVGT_body.cfweeds.cf I created coaching.cf and tsa-list.cf - they are basically one (well two technically) liners. If I need to cut back I'm not sure where to start - the SARE rules are the ones listed on rulesemporium.com - the FVGT are on http://www.exit0.us/index.php?pagename=RulesDuJourRuleSets. Of the rest, only bogus-virus-warnings.cf is of any substantial size (and still 1/3 the size of the largest SARE rule). Should v310.pre be removed if v312.pre is also there? Thanks, Josh
