Re: very basic SA-Learn performance question: is 90 seconds or so per token really, really slow or roughly normal? NYTProf results TxRep.pm 1720440 vs 1651114

[David Gessel]

FreeBSD is currently installing TxRep.pm rev 1651114 from Jan 12 15:17:46 2015 
(it is the only revision that has only whitespace differences, all leading 
padding, there are code differences between installed and 1650327 (previous) 
and 1678017 (next).  The most recent is 1720440 from Dec 16 20:23:02 2015.

https://svn.apache.org/viewvc/spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/TxRep.pm

TxRep.pm 1651114 emits error messages as below, it would seem for each message:

Use of uninitialized value $msgscore in addition (+) at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/TxRep.pm line 1415.
Use of uninitialized value $msgscore in subtraction (-) at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/TxRep.pm line 1415.

to be displayed as reported in bug 7164 
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7164

The fix is to use 1678017, though work seems to continue through 1720440.  
However, 1720440 seems to cause massive performance issues.

== With FreeBSD installed TxRep.pm (1651114) ==

# sa-learn --clear
# sa-learn --dump
ERROR: Bayes dump returned an error, please re-run with -D for more information

(folder has 236 messages)

# perl -T -d:NYTProf /usr/local/bin/sa-learn --spam --no-sync 
/mail/blackrosetech.com/gessel/.ManJunk/{cur,new}
Subroutine NetAddr::IP::STORABLE_freeze redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 365.
Subroutine NetAddr::IP::STORABLE_thaw redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 377.
Subroutine NetAddr::IP::STORABLE_freeze redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 365.
Subroutine NetAddr::IP::STORABLE_thaw redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 377.
Use of uninitialized value $msgscore in addition (+) at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/TxRep.pm line 1415.
Use of uninitialized value $msgscore in subtraction (-) at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/TxRep.pm line 1415.


Use of uninitialized value $msgscore in addition (+) at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/TxRep.pm line 1415.
Use of uninitialized value $msgscore in subtraction (-) at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/TxRep.pm line 1415.
Learned tokens from 234 message(s) (234 message(s) examined)
locker: error accessing /var/amavis/.spamassassin/bayes.lock: No such file or 
directory at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Locker/UnixNFSSafe.pm line 190.
locker: safe_unlock: lock on /var/amavis/.spamassassin/bayes.lock was lost due 
to expiry at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Locker/UnixNFSSafe.pm line 219.

# sa-learn --dump magic
0.000  0  3  0  non-token data: bayes db version
0.000  0234  0  non-token data: nspam
0.000  0  0  0  non-token data: nham
0.000  0  31174  0  non-token data: ntokens
0.000  0 1498800507  0  non-token data: oldest atime
0.000  0 1509734778  0  non-token data: newest atime
0.000  0 1509879373  0  non-token data: last journal sync atime
0.000  0  0  0  non-token data: last expiry atime
0.000  0  0  0  non-token data: last expire atime delta
0.000  0  0  0  non-token data: last expire reduction 
count

Profiling results for sa-learn: 
http://www.blackrosetech.com/insecure/nytprof-txrep-1651114/
732 seconds (3.1 seconds/message), 1.64 seconds in TxRep.pm (pretty trivial), 
mostly in the check_reputation subroutine
TxRep.pm profiling results 
http://www.blackrosetech.com/insecure/nytprof-txrep-1651114/Mail-SpamAssassin-Plugin-TxRep-pm-171-line.html

(side note, this maxed out one of 8 physical cores for the duration of the run, 
mostly at 100%)


== With latest TxRep.pm (1720440) ==

# sa-learn --clear
# sa-learn --dump
ERROR: Bayes dump returned an error, please re-run with -D for more information

(Same folder, 236 messages)

# perl -T -d:NYTProf /usr/local/bin/sa-learn --spam --no-sync 
/mail/blackrosetech.com/gessel/.ManJunk/{cur,new}
Subroutine NetAddr::IP::STORABLE_freeze redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 365.
Subroutine NetAddr::IP::STORABLE_thaw redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 377.
Subroutine NetAddr::IP::STORABLE_freeze redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 365.
Subroutine NetAddr::IP::STORABLE_thaw redefined at 
/usr/local/lib/perl5/site_perl/mach/5.24/NetAddr/IP.pm line 377.
Learned tokens from 234 message(s) (234 message(s) examined)
locker: error accessing /var/amavis/.spamassassin/bayes.lock: No such file or 
directory at 
/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Locker/UnixNFSSafe.pm line 190.
locker: safe_unlock: lock on /var/amavis/.spamassassin/bayes.lock was lost due 

Re: Rule updates?

[Bowie Bailey]

On 11/6/2017 11:29 AM, Merijn van den Kroonenberg wrote:

I saw some messages on the list indicating that rule updates were going
to resume starting about a week ago.  I haven't heard anything since and
still have not seen any updates.  What is the current status?

Its a work in progress, there was some feedback and some changes which had
to be made. Any change requires a couple days to propagate through the
masscheck system.

Hopefully we have a working update system soon, but I would expect still
at least a week to iron out some final things and probably another round
of testing before going live.

But all in all its really going forward now.


No problem.  Since there was no announcement of the delay (that I was 
able to find) and no one else seemed to be mentioning it at all, I just 
wanted to make sure there wasn't some problem on my end preventing the 
updates from coming through.


--
Bowie

Re: Rule updates?

[Merijn van den Kroonenberg]

> I saw some messages on the list indicating that rule updates were going
> to resume starting about a week ago.  I haven't heard anything since and
> still have not seen any updates.  What is the current status?

Its a work in progress, there was some feedback and some changes which had
to be made. Any change requires a couple days to propagate through the
masscheck system.

Hopefully we have a working update system soon, but I would expect still
at least a week to iron out some final things and probably another round
of testing before going live.

But all in all its really going forward now.

>
> --
> Bowie
>

Rule updates?

[Bowie Bailey]

I saw some messages on the list indicating that rule updates were going 
to resume starting about a week ago.  I haven't heard anything since and 
still have not seen any updates.  What is the current status?


--
Bowie

Re: Sending spam forwarded by backup MX directly to primary server spam folder

[Davide Marchi]

Il 2017-11-05 16:26 Matus UHLAR - fantomas ha scritto:
[..]


Is there any method to tell "primary server spamassassin"  
automatically move spam (forwarded by backup MX)  to (primary) spam 
folder?


SA does NOT move mail, it only scores mail.


..sorry, I expressed myself badly


However, you can tell your server to:

- not run SA when mail is received from backup MX
  (I assume you alweays run SA otherwise)

- move mail to spam folder when it's scored as spam
  (I think you have this done already)



Do you want to say that if I stop Spamassassin on Backup MX,
then the spam message coming from the backup server would it be marked 
as spam by the primary and then given in meal to the Sieve filter 
(Colombo Egg..)?



Thank you for helping to understand these elementary steps! :-)


Davide







--
cosmogoniA 
n o p r o v a r e n o f a r e o n o n f a r e n o n c e p r o v a r e