Some mail seems to hog spamd process

[Andreas Pettersson]

Hi.

Since yesterday I am having problem with spamd processes hogging cpu. 
All is fine until suddenly spamd keeps using 95% cpu forever. I noticed 
that bayes.lock also contains the pid of the hogging process. After some 
minutes I kill the pid and removes bayes.lock by hand, but it only takes 
a few minutes until the situation is the same again. I tailed the log 
trying to find some answers but only found


Sat Sep 23 12:50:25 2006 [13787] info: spamd: connection from localhost 
[127.0.0.1] at port 52807
Sat Sep 23 12:50:25 2006 [13787] info: spamd: checking message 
<[EMAIL PROTECTED]> for nobody:58


Does anyone have an idea on how to solve this?

Regards,
Andreas

Re: Some mail seems to hog spamd process

[Andreas Pettersson]

I have completely missed the recent thread "SA increasing load average a 
lot and spams getting through", which seems to reflect exactly the same 
problem I'm having.
For completeness I use SA 3.1.5 and haven't changed any cf the last few 
days.
Theres absolute not any high volume of mail. Plenty of time to process 
one mail at a time.


Regards,
Andreas


Andreas Pettersson wrote:


Hi.

Since yesterday I am having problem with spamd processes hogging cpu. 
All is fine until suddenly spamd keeps using 95% cpu forever. I 
noticed that bayes.lock also contains the pid of the hogging process. 
After some minutes I kill the pid and removes bayes.lock by hand, but 
it only takes a few minutes until the situation is the same again. I 
tailed the log trying to find some answers but only found


Sat Sep 23 12:50:25 2006 [13787] info: spamd: connection from 
localhost [127.0.0.1] at port 52807
Sat Sep 23 12:50:25 2006 [13787] info: spamd: checking message 
<[EMAIL PROTECTED]> for nobody:58


Does anyone have an idea on how to solve this?

Regards,
Andreas

Re: bayes sync is hogging cpu (was: Some mail seems to hog spamd process)

[Andreas Pettersson]

Hi, me again ;)

I'm pretty confident that the hogging occurs when SA is trying to sync 
the bayes. The bayes_journal is cleared exactly when the hogging begins. 
And when I run sa-learn --sync I get the very same hogging effect.


The permissions seems ok, doesn't it?

-rw---  1 spamd  wheel20 Sep 23 13:28 bayes.lock
-rw---  1 spamd  wheel  2760 Sep 23 13:28 bayes_journal
-rw-r--r--  1 spamd  wheel  83755008 Sep 23 13:28 bayes_seen
-rw---  1 spamd  wheel  83853312 Sep 23 13:28 bayes_toks


Regards,
Andreas

Re: sa-learn question

[Matt Kettler]

Russell Jones wrote:
> If I have multiple sa-learn processes going at the same time, can that
> corrupt the database and/or cause some other problem that I don't want
> to happen? Or is it safe to have the following in crontab for example:
>  
> @daily sa-learn --spam
> /home/eggycrew/imap/eggycrew.com/rjones/Maildir/.INBOX.spam
> @daily sa-learn --ham /home/eggycrew/imap/eggycrew.com/rjones/Maildir/cur
> @daily sa-learn --ham /home/eggycrew/imap/eggycrew.com/rjones/Maildir/new
Well, nothing bad will happen, but they'll all effectively get run one
at a time. Since only one process can have the R/W lock on the bayes DB,
one of them will get the lock and the others will go to sleep waiting
for the lock to be released.

Re: Problem with user_white_list

[Matt Kettler]

Theo Van Dinter wrote:
> On Fri, Sep 22, 2006 at 04:01:20AM -0400, Matt Kettler wrote:
>   
 The moral here is NEVER use whitelist_from. 
 
>>> ...does this indicate that whitelist_from should be obsoleted?
>>>   
>> should, yes.. will be, probably not.
>> 
>
> Well, there is a need and are uses for whitelist_from, specifically when the
> other options aren't available.  Have a customer who sends you mail, but they
> don't have proper rDNS setup nor SPF nor ... ?
>   
I do agree with your point, and that's the reason why I said it probably
will not be obsoleted.

However, the guy with no rDNS nor SPF isn't very likely to be able to
send mail to very many places at all. Now that major ISPs (ie: AOL) are
blocking servers with no RDNS, it's only a matter of time before this
becomes standard practice and he won't be able to send mail anywhere.

Also, I personally view ANY spamassassin whitelisting feature as a
measure of last resort. It's generally better to whitelist by
configuring your tools to not call SA in the first place. You have more
reliable envelope information, AND you gain CPU usage benefits.

Re: Problem with user_white_list

[Benny Pedersen]

On Sat, September 23, 2006 15:20, Matt Kettler wrote:
> Also, I personally view ANY spamassassin whitelisting feature as a
> measure of last resort. It's generally better to whitelist by
> configuring your tools to not call SA in the first place. You have more
> reliable envelope information, AND you gain CPU usage benefits.

change score on whitelist_from solves most here

else ham is not learned from local users

-- 
"This message was sent using 100% recycled spam mails."

Assign a score to a tflag

[Lists]

Title: Assign a score to a tflag






Hello,


I have integrate some spamassassin rule (copied from /usr/share/spamassassin) in order to use some blacklists in the world.

As example i copied this rule:


header RCVD_IN_SBL  eval:check_rbl_sub('sblxbl', '127.0.0.2')

describe RCVD_IN_SBL    Received via a relay in Spamhaus SBL

tflags RCVD_IN_SBL  net


Is there a way, automatically, to assign a score (as example 20.0 so it is automatically detected as Spam) if the conditions is true? (Perhaps the tflags has to be "net"?)

Thank you in advance for your kind interest, cheers!



Francesco

Re: SA increasing load average a lot and spams getting through

[Payal Rathod]

On Fri, Sep 22, 2006 at 10:44:11AM -0700, John Goubeaux wrote:
> you might also want to check:
> 
> http://spamassassin.apache.org/full/3.1.x/dist/doc/Mail_SpamAssassin_Conf.html#miscellaneous_options
> 
> eg  lock_method   I had the same bayes files errors and set this to 
> lock_method flock   and think i have alleviated the problems.

This has really helped a LOT.

With warm regards,
-Payal

Re: Assign a score to a tflag

[Loren Wilton]

Assign a score to a tflag> header RCVD_IN_SBL 
eval:check_rbl_sub('sblxbl', '127.0.0.2')

describe RCVD_IN_SBLReceived via a relay in Spamhaus SBL
tflags RCVD_IN_SBL  net


Is there a way, automatically, to assign a score (as example 20.0 so it is 
automatically
detected as Spam) if the conditions is true? (Perhaps the tflags has to be 
"net"?)


scoreRCVD_IN_SBL  20 

Re: Any plugins that scan looking for phone numbers? (URICountry)

[Robert Nicholson]

It looks like I get by the URICountry plugin but I have a question  
about that.


why does it use IP::Country::Fast instead of IP::Country::Medium  
since it's not doing the lookup by ip but by URI which is by name  
more often than ip address.


On Sep 22, 2006, at 7:53 PM, Robert Nicholson wrote:

I have a need to isolate any mail that contains a UK phone number.  
I would expect there's a plugin that does something already?

Re: Problem with user_white_list

[Bookworm]

Matt Kettler wrote:

Theo Van Dinter wrote:
  

On Fri, Sep 22, 2006 at 04:01:20AM -0400, Matt Kettler wrote:
  

The moral here is NEVER use whitelist_from. 

  

...does this indicate that whitelist_from should be obsoleted?
  


should, yes.. will be, probably not.

  

Well, there is a need and are uses for whitelist_from, specifically when the
other options aren't available.  Have a customer who sends you mail, but they
don't have proper rDNS setup nor SPF nor ... ?
  


I do agree with your point, and that's the reason why I said it probably
will not be obsoleted.

However, the guy with no rDNS nor SPF isn't very likely to be able to
send mail to very many places at all. Now that major ISPs (ie: AOL) are
blocking servers with no RDNS, it's only a matter of time before this
becomes standard practice and he won't be able to send mail anywhere.

Also, I personally view ANY spamassassin whitelisting feature as a
measure of last resort. It's generally better to whitelist by
configuring your tools to not call SA in the first place. You have more
reliable envelope information, AND you gain CPU usage benefits.
  
Unfortunately, I've never had to use whitelist_from for RDNS/SPF 
problems.  What I've had to use it for is that shipping companies 
(Customs Brokers, freight forwarders, warehousers) tend to write 
_everything_ in all caps, no matter what.   Forms for the government, 
online databases, you name it, it's one case, and that's upper.   
Needless to say, that causes SA to blow up on it, and claim that all of 
the emails going in and out are spam.  Since I _want_ caps to generally 
set off SA, I end up with a list of shipping related companies that I 
whitelist_from by default.


BW

RCVD_IN_WHOIS_INVALID

[Kenneth Porter]

2.2 RCVD_IN_WHOIS_INVALID  RBL: CompleteWhois: sender on invalid IP block
  [65.119.30.206 listed in 
combined-HIB.dnsiplists.completewhois.com]


I just got an order confirmation from Newegg and it got a big score boost 
of 2.2 from this rule. What does this rule mean? I ran the address through 
the whois form at http://arin.net/ and it's listed in Quest's block. Is 
this complaining that there's no more detailed information for the exact 
address?

Re: RCVD_IN_WHOIS_INVALID

[Magnus Holmgren]

On Saturday 23 September 2006 22:50, Kenneth Porter took the opportunity to 
say:
>  2.2 RCVD_IN_WHOIS_INVALID  RBL: CompleteWhois: sender on invalid IP block
>[65.119.30.206 listed in
> combined-HIB.dnsiplists.completewhois.com]
>
> I just got an order confirmation from Newegg and it got a big score boost
> of 2.2 from this rule. What does this rule mean? I ran the address through
> the whois form at http://arin.net/ and it's listed in Quest's block. Is
> this complaining that there's no more detailed information for the exact
> address?

See 
http://cwhois0.completewhois.com/cgi-bin/dbcheck-invalidipwhois.cgi?IP=65.119.30.206

Apparently the listing, which was imported from rfc-ignorant.org two years 
ago, is obsolete.

-- 
Magnus Holmgren[EMAIL PROTECTED]
   (No Cc of list mail needed, thanks)


pgpRCC0f1i58N.pgp
Description: PGP signature

[off-topic] Recommended Commercial DNS Services?

[Rob McEwen]

Recommended Commercial DNS Services?

 

I’m looking for suggestions for reliable outsourced
DNS services where the servers aren’t overloaded, the prices are
reasonable, and the service & control panels are tops.

 

Any suggestions?

 

Rob McEwen

PowerView Systems

[EMAIL PROTECTED]

 

SA gone mad, times out and stucks

[Jürgen Herz]

Hello!

I'm running Exim together with spamc/spamd on my box for months now
without problems. But a week ago many spams begun to show up in my
Inbox, so I investigated what's wrong. Until recently most spamd.log
entries looked like this:

info: spamd: got connection over /var/run/spamd.sock
info: spamd: setuid to Debian-exim succeeded
info: spamd: processing message <[EMAIL PROTECTED]> for Debian-exim:106
info: spamd: clean message (-2.4/5.0) for Debian-exim:106 in 5.7
seconds, 4793 bytes.
info: spamd: result: . -2 - SOME_CHECKS
scantime=5.7,size=4793,user=Debian-exim,uid=106,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=/var/run/spamd.sock,mid=<[EMAIL
 PROTECTED]>,bayes=0,autolearn=no


But now soon after restarting Spamassassin, Exim reports "spamd took
more than 60 secs to run" (and thus the connection times out and Exim
doesn't sort out spams anymore). And from this point on spamd.log
doesn't contain any new entries.
Before discontinuation of service, log entries contain warns like

warn: bayes: cannot open bayes databases
/var/spool/exim4/.spamassassin/bayes_* R/W: lock failed: File exists
(Often repeated two or three times for each mail.)

warn: bayes: expire_old_tokens: child processing timeout at
/usr/sbin/spamd line 1086.
(Spamd then takes very long to scan a mail:
info: spamd: clean message (0.0/5.0) for Debian-exim:106 in 305.0
seconds, 3781 bytes.)

These mails are neither big nor is the machine under heavy load. Other
messages (and all formerly) of same (and bigger) size take about five
seconds.

Netstat reports quite a few spamd.pid and spamd.child (some with
spamd.sock) though max-children is 2.

I'm using Spamassassin 3.1.3 from Debian backports for stable (PPC).
It's started with
--create-prefs --max-children 2 --syslog=/var/log/spamd.log
--helper-home-dir --socketpath=/var/run/spamd.sock


Any thoughts what's wrong?

Regards,
Jürgen

Re: Assign a score to a tflag

[jdow]

From: "Loren Wilton" <[EMAIL PROTECTED]>

Assign a score to a tflag> header RCVD_IN_SBL 
eval:check_rbl_sub('sblxbl', '127.0.0.2')

describe RCVD_IN_SBLReceived via a relay in Spamhaus SBL
tflags RCVD_IN_SBL  net


Is there a way, automatically, to assign a score (as example 20.0 so it is 
automatically
detected as Spam) if the conditions is true? (Perhaps the tflags has to be 
"net"?)


scoreRCVD_IN_SBL  20


At that point he should move the SBLXBL test outside of SpamAssassin
and save machine load. Experience suggests blocking on the basis of
one BL is unwise.

{^_^}

Re: Problem with user_white_list

[Matt Kettler]

Benny Pedersen wrote:
> On Sat, September 23, 2006 15:20, Matt Kettler wrote:
>   
>> Also, I personally view ANY spamassassin whitelisting feature as a
>> measure of last resort. It's generally better to whitelist by
>> configuring your tools to not call SA in the first place. You have more
>> reliable envelope information, AND you gain CPU usage benefits.
>> 
>
> change score on whitelist_from solves most here
>
> else ham is not learned from local users
>
>   
Eh? How does changing the score solve anything? You do know that the
whitelist score doesn't affect the autolearner, right?

unsubscribe

[Chris Mills (Chrysalis)]

unsubscribe

Re: unsubscribe

[Evan Platt]

At 08:43 PM 9/23/2006, Chris Mills (Chrysalis) wrote:

unsubscribe


list-unsubscribe: 

Re: SA gone mad, times out and stucks

[Loren Wilton]

warn: bayes: expire_old_tokens: child processing timeout at
/usr/sbin/spamd line 1086.
(Spamd then takes very long to scan a mail:
info: spamd: clean message (0.0/5.0) for Debian-exim:106 in 305.0
seconds, 3781 bytes.)


The child is trying to run a Bayes expire, apparently on a large Bayes 
database that hasn't had a successful expiry run in some time.  This attempt 
to process the Bayes database is probably taking over 300 seconds, and the 
child is being timed out and killed by something.  As a result of being 
killed, it never finished the Bayes expire processing.  So the next child 
tries to do the same thing, gets timed out and killed, the nex child tries 
to do the same thing...


Run a manual Bayes expire run and it will probably clean up your problems. 
If this sort of problem starts to reoccur you might consider turning off 
bayes auto expire and setting up a cron run to do it once a day or so.  (Or 
more often, depending on your mail volume.)


   Loren

Re: SA gone mad, times out and stucks

[jdow]

From: "Loren Wilton" <[EMAIL PROTECTED]>


warn: bayes: expire_old_tokens: child processing timeout at
/usr/sbin/spamd line 1086.
(Spamd then takes very long to scan a mail:
info: spamd: clean message (0.0/5.0) for Debian-exim:106 in 305.0
seconds, 3781 bytes.)


The child is trying to run a Bayes expire, apparently on a large Bayes database that 
hasn't had a successful expiry run in some time.  This attempt to process the Bayes 
database is probably taking over 300 seconds, and the child is being timed out and 
killed by something.  As a result of being killed, it never finished the Bayes expire 
processing.  So the next child tries to do the same thing, gets timed out and killed, 
the nex child tries to do the same thing...


Run a manual Bayes expire run and it will probably clean up your problems. If this sort 
of problem starts to reoccur you might consider turning off bayes auto expire and 
setting up a cron run to do it once a day or so.  (Or > more often, depending on your 
mail volume.)


Not that I necessarily disbelieve the need for such purges of history
in the BAYES database I am moved to observe that he who ignores history
is doomed to repeat history's mistakes.

{^_^} 

Re: bayes sync is hogging cpu

[Andreas Pettersson]

Ok, more information here.

I found in spamd.log this line when the problem started:
Fri Sep 22 19:55:22 2006 [74581] warn: bayes: expire_old_tokens: child 
processing timeout at /usr/local/bin/spamd line 1082


which was followed by lots of these:
Fri Sep 22 19:55:52 2006 [74581] warn: bayes: cannot open bayes 
databases /usr/local/share/spamassassin/bayes/bayes_* R/W:

lock failed: File exists

In an attempt to find what's wrong I changed bayes_learn_to_journal to 
1. It didn't help, but at least I got rid of the 'lock failed: File 
exist' error messages in spamd.log and bayes also keeps working. For the 
moment I have a script that checks for bayes.lock existance and kills 
the hogging process and removes the lock file. It runs every minute..



I have tried change lock_method to flock, problem still there (but with 
a new lock file name).
I also tried a sa-learn --force-expire. It took about 30 sec to 
complete. It didn't solve my problem either.



Any ideas of what might be wrong?

Regards,
Andreas