Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
Yes, that does look like it. If that hasn't made it into the standard Fedora repo by the time of my next scheduled update I'll pull it it from Testing - I've got enough other stuff I need to deal with right now without adding in a 3.3.2-3.4.0 conversion. Final follow-up === Yesterday my weekly Fedora 20 update delivered SA 3.4.0 and updated Net::DNS back 0.75 from last week's downgrade to 0.72. I've just been running tests using my standard SA 3.3.2 configuration and, apart from a tweak or to to local rule scores, its all running well and producing the expected results on my test corpus. Many thanks to the info provided by this list and Karsten's prodding. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Mon, 2014-06-02 at 02:41 +0200, Karsten Bräckelmann wrote: If that is the culprit, the easiest, fastest and most painless way of getting a fully functional SA back, is to revert the recent Perl Net::DNS upgrade. Yes, I can now confirm that the problem was the recent upgrade of Net::DNS 0.72 to 0.75. After downgrading perl-Net-DNS-0.75-1.fc20.i686 to perl-Net-DNS-0.72-6.fc20.i686 spamd 3.3.2 is now working as expected. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sun, 2014-06-01 at 03:01 +0200, Karsten Bräckelmann wrote: That error message rings a bell. Will (aragonx?) posted that line very recently, and updated the thread himself just today, pointing to a RH / Fedora 20 bugzilla report for its SA 3.3.2 package, related to Perl Net::DNS 0.76 (once available). https://bugzilla.redhat.com/show_bug.cgi?id=1096405 Comment 5 also mentions an issue with Perl Net::DNS 0.75, which is the exact version the package upgrade pulled in on your system before this started. The report matches the first insecure dependency above. Yes, that does look like it. If that hasn't made it into the standard Fedora repo by the time of my next scheduled update I'll pull it it from Testing - I've got enough other stuff I need to deal with right now without adding in a 3.3.2-3.4.0 conversion. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sun, 2014-06-01 at 09:13 +0100, Martin Gregorie wrote:
On Sun, 2014-06-01 at 03:01 +0200, Karsten Bräckelmann wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=1096405
Comment 5 also mentions an issue with Perl Net::DNS 0.75, which is the
exact version the package upgrade pulled in on your system before this
started. The report matches the first insecure dependency above.
Yes, that does look like it. If that hasn't made it into the standard
Fedora repo by the time of my next scheduled update I'll pull it it from
Testing - I've got enough other stuff I need to deal with right now
without adding in a 3.3.2-3.4.0 conversion.
Without moving from SA 3.3.2 (current Fedora) to 3.4.0, regardless of it
getting promoted up...
If that is the culprit, the easiest, fastest and most painless way of
getting a fully functional SA back, is to revert the recent Perl
Net::DNS upgrade.
Just manually download and install the previous Net::DNS package.
Besides likely fixing your issue, it would be nice as a confirmation for
the list.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Tue, 2014-05-27 at 02:48 +0200, Karsten Bräckelmann wrote:
On Sat, 2014-05-24 at 12:34 +0100, Martin Gregorie wrote:
On Sat, 2014-05-24 at 05:04 +0200, Karsten Bräckelmann wrote:
LATER: This morning I reran some failing examples after rebooting the
test machine. No change, so I tried a few stripped-down runs, i.e. I
started spamd via a test script that uses systemctl to start or stop it
and then used spamc testmessage to exercise it while I played round
with spamc options. In the course of this I noticed a strange effect:
The FIRST message after restarting spamd never has X-Spam headers, but
the second and subsequent ones do have X-Spam headers.
I'm running these versions:
$ spamd --version
SpamAssassin Server version 3.3.2
running on Perl 5.18.2
with SSL support (IO::Socket::SSL 1.955)
with zlib support (Compress::Zlib 2.062)
$ spamc --version
SpamAssassin Client version 3.2.4
Should the spamc/spamd version mismatch have any bad effects?
No, there should be absolutely no problems, since spamc/d are using a
protocol that hasn't even added features since 3.2, let alone changed in
an incompatible way.
It does indicate a real problem with your package management or custom
builds, though.
The only SA package I have installed is spamassassin.i686 3.3.2-56.fc20
which came from atrpms. This is odd, since all the uninstalled packages
(spamass-milter, spamass-milter-postfix, spamassassin-FuzzyOcr,
spamassassin-iXhash2, spambayes, spampd, spamprobe.i686) are from Fedore
repositories as you'd expect. Removing and reinstalling the spamassassin
package has had no effect.
I'll take this up with RedHat next week and see if I can find out why
they no longer provide the main spamassassin package in their
repository.
A quick googlin' brings up spamassassin 3.3.2-18.fc20 for Fedora 20, in
a single package shipping both spamc and spamd in /usr/bin.
After deleting and reinstalling (yum install spamassassin) I now have
spamassassin.i686 3.3.2-18.fc20 installed but with no apparent effect:
it is running these versions:
# spamd --version
SpamAssassin Server version 3.3.2
running on Perl 5.18.2
with SSL support (IO::Socket::SSL 1.955)
with zlib support (Compress::Zlib 2.062)
# spamc --version
SpamAssassin Client version 3.2.4
exactly as before. Yum says it knows about these spamassassin package:
Installed Packages
spamassassin.i686 3.3.2-18.fc20 @updates
Available Packages
spamassassin.i686 3.3.2-56.fc20 atrpms
which at least explains why the atrpms package was installed rather than
the Fedora @updates one.
The only problem is as before
$ start sa# private script which uses systemctl start spamassassin
# which is running under root as expected
$ ps -ef |grep spamd
root 23426 1 6 17:28 ?00:00:03 /usr/bin/spamd
--pidfile /var/run/spamd.pid -d -c -m5 -H
root 23427 23426 0 17:28 ?00:00:00 spamd child
root 23428 23426 0 17:28 ?00:00:00 spamd child
kiwi 23449 23142 0 17:29 pts/100:00:00 grep --color=auto spamd
$ testsa data/sale57.txt
Checking data/sale57.txt
Checked 1 messages in 0 secs, 0 mS/message
$ testsa data/sale57.txt
Checking data/sale57.txt
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on
zappa.gregorie.lan
X-Spam-Flag: YES
X-Spam-Level: ***
X-Spam-Status: Yes, score=7.9 required=6.0 tests=HTML_IMAGE_RATIO_04,
HTML_MESSAGE,MG_MONEY,MG_PRICE,MG_PRODUCT,MG_SALE,MG_SALESOFFER,MG_SALESPAM,
MG_SPAMREF,RP_MATCHES_RCVD,T_DKIM_INVALID,T_END_FUTURE_EMAILS,
UNPARSEABLE_RELAY autolearn=no version=3.3.2
Checked 1 messages in 0 secs, 0 mS/message
$ sa stop
As you can see, spamc/spamd from @updates does exactly the same as the
one from atrpms: the first message passed by spamc after spamd is
started apparently does nothing but the second and subsequent ones are
scanned.
The testsa script looks like this:
$ cat testsa
#!/bin/bash
#
#help
# Syntax: testsa [-x] testmessage.
# Function: Run one or more messages through spamc
# The output is filtered to contain only headers
# added by Spamassassin
# Options: -x show the whole message
#end
if [ $1 == '-?' ]
then
script_help testsa
exit 1
fi
state=$(spamdstatus)
if [ $state == 'spamd is stopped' ]
then
sudo systemctl start spamassassin.service
fi
if [ $1 == '-x' ]
then
filter=no
shift
else
filter=yes
fi
tcount=0
starttime=$(date +%s)
for s
do
echo Checking $s
echo
if [ $filter == 'yes' ]
then
spamc --max-size=100 $s | gawk '
BEGIN { tag=0 }
/^X-Spam/ { tag=1; print; next }
/^ / || /^\t/ { if (tag==1) { print } next }
{ tag = 0 }
'
else
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-31 at 17:39 +0100, Martin Gregorie wrote:
On Tue, 2014-05-27 at 02:48 +0200, Karsten Bräckelmann wrote:
A quick googlin' brings up spamassassin 3.3.2-18.fc20 for Fedora 20, in
a single package shipping both spamc and spamd in /usr/bin.
After deleting and reinstalling (yum install spamassassin) I now have
spamassassin.i686 3.3.2-18.fc20 installed but with no apparent effect:
it is running these versions:
# spamd --version
SpamAssassin Server version 3.3.2
running on Perl 5.18.2
with SSL support (IO::Socket::SSL 1.955)
with zlib support (Compress::Zlib 2.062)
# spamc --version
SpamAssassin Client version 3.2.4
$ which -a spamc
The Fedora spamassassin 3.3.2 RPM you just installed does ship spamc
in /usr/bin. The version mismatch persists, so there is another spamc
from a different source early in your $PATH.
Use 'rpm -qf file' to find out which package it belongs to, if any.
In either case, packaged or built from source, we're chasing something
in your environment or system which you are obviously unaware of.
Assuming you are the only one administrating that system: Without
pointing fingers, the system has to be considered dirty.
Given that still unexplained version mismatch, it is not unlikely there
are bad installations of Perl libraries, too. Might be time to install
the system fresh...
The testsa script looks like this:
state=$(spamdstatus)
if [ $state == 'spamd is stopped' ]
then
sudo systemctl start spamassassin.service
fi
Most likely unrelated, though this code might potentially mess with
spamd unnoticed. I'd get rid of that part while debugging the issue, or
at least make it clearly print a warning.
Obviously something is wrong, but should this be raised on the Fedora
Bugzilla or would it be better raised within the SA project?
I suggest to first try to reproduce the issue on a fresh installed
system.
Apart from that, the issue surfaced after a system upgrade, without any
SA code changed. Makes the distro the more likely place for reporting.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-31 at 20:15 +0200, Karsten Bräckelmann wrote: $ which -a spamc 'locate spamc' turned up a copy of spamc 3.2.4 in /usr/local/bin dated 2008. I can't remember how it might have got there since I've only ever installed SA from the Fedora repo. Anyway, that is gone now and both spamc and spamd are at version 3.3.2 The only copy of spamd is 3.3.2 and AFAICT there have never been additional copies of it or spamassassin - also 3.3.2 Given that still unexplained version mismatch, it is not unlikely there are bad installations of Perl libraries, too. Might be time to install the system fresh... That happened a few weeks back when I did a clean install of Fedora 20. This replaced all executables that aren't in /usr/local/bin: the /usr/local directory is a symlink to /home/local and home is a separate partition and is the only one on this system that isn't reformatted during a clean install. The testsa script looks like this: state=$(spamdstatus) if [ $state == 'spamd is stopped' ] then sudo systemctl start spamassassin.service fi Most likely unrelated, though this code might potentially mess with spamd unnoticed. I'd get rid of that part while debugging the issue, or at least make it clearly print a warning. This script looks like this: #!/bin/bash sudo systemctl status spamassassin.service 21 /dev/null if [ $? -eq 0 ] then reply='spamd is running' else reply='spamd is stopped' fi echo $reply So I don't see how it can mess with spamd. In any case the first time failure happens regardless of whether testsa is allowed to start and stop spamd of it it is started separately. The only difference is that if testsa does the start/stop it always fails since every test it submits is the first spamc request after a restart, while if spamd is started separately, only the first test fails to scan the message: the rest are OK until spamd is stopped and restarted. Apart from that, the issue surfaced after a system upgrade, without any SA code changed. Makes the distro the more likely place for reporting. Yeah, I'll raise a bug with Fedora and spare this mailing list from more bandwidth use. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-31 at 23:07 +0100, Martin Gregorie wrote:
On Sat, 2014-05-31 at 20:15 +0200, Karsten Bräckelmann wrote:
The testsa script looks like this:
state=$(spamdstatus)
if [ $state == 'spamd is stopped' ]
then
sudo systemctl start spamassassin.service
fi
Most likely unrelated, though this code might potentially mess with
spamd unnoticed. I'd get rid of that part while debugging the issue, or
at least make it clearly print a warning.
This script looks like this:
So I don't see how it can mess with spamd.
By messing with I mean code that potentially (re)starts spamd, without
us knowing. That is an absolute no-go when debugging or tracking down
bugs.
In any case the first time
failure happens regardless of whether testsa is allowed to start and
stop spamd of it it is started separately. The only difference is that
if testsa does the start/stop it always fails since every test it
submits is the first spamc request after a restart, while if spamd is
started separately, only the first test fails to scan the message: the
rest are OK until spamd is stopped and restarted.
Actually, thinking about that (re)starting of spamd, and only the first
attempt failing:
I haven't really used systemd yet, but one fundamental design decision
is, that systemd itself takes care about sockets and stuff, returning
early and asynchronously lets the service complete starting up in the
background.
This might explain what you are observing, the first spamc call failing,
if you do run spamc close after spamd service starting. Can you still
reproduce the issue, if you wait a minute before running spamc?
Did you ever verify spamd was running, or did you just run your usual
test script?
Also, IIRC you never mentioned the spamc exit code. See the man-page for
details, spamc differentiates between a lot of network and other error
conditions.
Generally, while debugging $product issues, you should first get complex
custom wrappers out of the equation. With spamc, a stripped down test
case is:
echo | spamc
echo $?
No complex wrapper, no spam needed. Generate minimal input. There will
always be at least an X-Spam-Version header, if it passed spamd. Get the
exit code in case of failure (no X-Spam header), compare with the
man-page.
Run once *immediately* after starting spamd as usual. Stop and restart
spamd as usual, wait, and try stripped down test case again.
For bonus-points, watch the logs for spamd claiming to be ready.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sun, 2014-06-01 at 01:21 +0200, Karsten Bräckelmann wrote: I haven't really used systemd yet, but one fundamental design decision is, that systemd itself takes care about sockets and stuff, returning early and asynchronously lets the service complete starting up in the background. That could easily fit this scenario. This might explain what you are observing, the first spamc call failing, if you do run spamc close after spamd service starting. Can you still reproduce the issue, if you wait a minute before running spamc? Did you ever verify spamd was running, or did you just run your usual test script? Yep. Thats implied with a separate spamd start: sometimes I check that its running (script wrapper round 'systemctl status spamassassin') but you can tell from inspection because the testsa script takes a lot longer to run if it has to start spamd. Also, IIRC you never mentioned the spamc exit code. See the man-page for details, spamc differentiates between a lot of network and other error conditions. =Here's the instant 'start spamd, run spamc, stop spamd test: $ sudo systemctl start spamassassin; echo $?; echo | spamc; echo $?; sudo systemctl stop spamassassin 0 0 $ ===and here's the version with a delay after starting: $ sudo systemctl start spamassassin; echo $?; sleep 30 0 $ echo | spamc; echo $? 0 $ sudo systemctl stop spamassassin $ =same again but with 2 messages and no delay $ sudo systemctl start spamassassin; echo $?; spamc x.txt; echo $?; spamc x.txt; echo $?; sudo systemctl stop spamassassin 0 To: example.com body text. . 0 To: example.com body text. . 0 as the last, but with a 2 sec delay between messages $ sudo systemctl start spamassassin; echo $?; spamc x.txt; echo $?; sleep 2; spamc x.txt; echo $?; sudo systemctl stop spamassassin 0 To: example.com body text. . 0 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on zappa.gregorie.lan X-Spam-Level: * X-Spam-Status: No, score=5.6 required=6.0 tests=MG_WRONG_DOMAIN,MISSING_DATE, MISSING_FROM,MISSING_MID,MISSING_SUBJECT,NO_RECEIVED,NO_RELAYS,TO_MALFORMED autolearn=no version=3.3.2 To: example.com body text. . 0 $ For bonus-points, watch the logs for spamd claiming to be ready. Here you go: Jun 1 01:07:41 zappa spamd[15831]: plugin: eval failed: Insecure dependency in connect while running with -T switch at /usr/lib/perl5/IO/Socket.pm line 115. Jun 1 01:07:42 zappa spamd[15831]: spamd: server started on port 783/tcp (running version 3.3.2) Jun 1 01:07:42 zappa spamd[15831]: spamd: server pid: 15831 Jun 1 01:07:42 zappa spamd[15831]: spamd: server successfully spawned child process, pid 15832 Jun 1 01:07:42 zappa spamd[15831]: spamd: server successfully spawned child process, pid 15833 Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: IS Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: II Jun 1 01:07:42 zappa spamd[15832]: spamd: connection from zappa.gregorie.lan [127.0.0.1] at port 39842 Jun 1 01:07:42 zappa spamd[15832]: spamd: setuid to kiwi succeeded Jun 1 01:07:42 zappa spamd[15832]: spamd: Insecure dependency in connect while running setuid at /usr/lib/perl5/IO/Socket.pm line 115, GEN9 line 9. Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: II Jun 1 01:07:44 zappa spamd[15832]: spamd: connection from zappa.gregorie.lan [127.0.0.1] at port 39843 Jun 1 01:07:44 zappa spamd[15832]: spamd: setuid to kiwi succeeded Jun 1 01:07:44 zappa spamd[15832]: spamd: processing message (unknown) for kiwi:1000 Jun 1 01:07:44 zappa spamd[15832]: spamd: clean message (5.6/6.0) for kiwi:1000 in 0.1 seconds, 31 bytes. Jun 1 01:07:44 zappa spamd[15832]: spamd: result: . 5 - MG_WRONG_DOMAIN,MISSING_DATE,MISSING_FROM,MISSING_MID,MISSING_SUBJECT,NO_RECEIVED,NO_RELAYS,TO_MALFORMED scantime=0.1,size=31,user=kiwi,uid=1000,required_score=6.0,rhost=zappa.gregorie.lan,raddr=127.0.0.1,rport=39843,mid=(unknown),autolearn=no Jun 1 01:07:44 zappa spamd[15831]: spamd: server killed by SIGTERM, shutting down I think the 2nd 'insecure dependency' at 01:07:42 is associated with spamd receiving the first message Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sun, 2014-06-01 at 01:20 +0100, Martin Gregorie wrote:
On Sun, 2014-06-01 at 01:21 +0200, Karsten Bräckelmann wrote:
For bonus-points, watch the logs for spamd claiming to be ready.
Here you go:
Jun 1 01:07:41 zappa spamd[15831]: plugin: eval failed: Insecure
dependency in connect while running with -T switch
at /usr/lib/perl5/IO/Socket.pm line 115.
Jun 1 01:07:42 zappa spamd[15832]: spamd: connection from
zappa.gregorie.lan [127.0.0.1] at port 39842
Jun 1 01:07:42 zappa spamd[15832]: spamd: setuid to kiwi succeeded
Jun 1 01:07:42 zappa spamd[15832]: spamd: Insecure dependency in
connect while running setuid at /usr/lib/perl5/IO/Socket.pm line 115,
GEN9 line 9.
Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: II
Jun 1 01:07:44 zappa spamd[15832]: spamd: connection from
zappa.gregorie.lan [127.0.0.1] at port 39843
Jun 1 01:07:44 zappa spamd[15832]: spamd: setuid to kiwi succeeded
Jun 1 01:07:44 zappa spamd[15832]: spamd: processing message (unknown)
for kiwi:1000
I think the 2nd 'insecure dependency' at 01:07:42 is associated with
spamd receiving the first message
If so, that would be the one passed back unprocessed, while the
following invocation works.
That error message rings a bell. Will (aragonx?) posted that line very
recently, and updated the thread himself just today, pointing to a RH /
Fedora 20 bugzilla report for its SA 3.3.2 package, related to Perl
Net::DNS 0.76 (once available).
https://bugzilla.redhat.com/show_bug.cgi?id=1096405
Comment 5 also mentions an issue with Perl Net::DNS 0.75, which is the
exact version the package upgrade pulled in on your system before this
started. The report matches the first insecure dependency above.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 12:34 +0100, Martin Gregorie wrote:
On Sat, 2014-05-24 at 05:04 +0200, Karsten Bräckelmann wrote:
LATER: This morning I reran some failing examples after rebooting the
test machine. No change, so I tried a few stripped-down runs, i.e. I
started spamd via a test script that uses systemctl to start or stop it
and then used spamc testmessage to exercise it while I played round
with spamc options. In the course of this I noticed a strange effect:
The FIRST message after restarting spamd never has X-Spam headers, but
the second and subsequent ones do have X-Spam headers.
I'm running these versions:
$ spamd --version
SpamAssassin Server version 3.3.2
running on Perl 5.18.2
with SSL support (IO::Socket::SSL 1.955)
with zlib support (Compress::Zlib 2.062)
$ spamc --version
SpamAssassin Client version 3.2.4
Should the spamc/spamd version mismatch have any bad effects?
No, there should be absolutely no problems, since spamc/d are using a
protocol that hasn't even added features since 3.2, let alone changed in
an incompatible way.
It does indicate a real problem with your package management or custom
builds, though.
The only SA package I have installed is spamassassin.i686 3.3.2-56.fc20
which came from atrpms. This is odd, since all the uninstalled packages
(spamass-milter, spamass-milter-postfix, spamassassin-FuzzyOcr,
spamassassin-iXhash2, spambayes, spampd, spamprobe.i686) are from Fedore
repositories as you'd expect. Removing and reinstalling the spamassassin
package has had no effect.
I'll take this up with RedHat next week and see if I can find out why
they no longer provide the main spamassassin package in their
repository.
A quick googlin' brings up spamassassin 3.3.2-18.fc20 for Fedora 20, in
a single package shipping both spamc and spamd in /usr/bin.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 05:04 +0200, Karsten Bräckelmann wrote: On Sat, 2014-05-24 at 03:28 +0100, Martin Gregorie wrote: On Sat, 2014-05-24 at 02:36 +0200, Karsten Bräckelmann wrote: I've looked down the list a couple of times and didn't see anything I thought would affect it due to a possibly bad assumption that this sort of error would be insulated from the underlying binary code by at least one layer of Perl library code. Plain spamassassin script works, spamc/d combination does not. Nope, this is not necessarily related to any Perl code. OK. Looking through the full list of upgraded packages, the two SELinux related ones stick out. In particular, since the client / server environment is affected, but not the standalone variant of calling the majority of shared SA Perl code. SELinux shouldn't be an issue as its running in Permissive, Targeted mode and it hasn't flagged up any warnings. By started it, do you mean starting the daemon simply for that test, or did you ping spamd after launching your dev test rig? Yes, I meant starting it. As I said, this problem surfaced on the box I use for SA rules development, actually a Lenovo laptop. Consequently SA only runs on it during a rules test: all my test scripts are written to start spamd, run the test and stop it. Clean room vs. production conditions... Yes, exactly. Nah. I was talking about the difference in your test (manually starting spamd) and the environment observed to fail (your dev test rig script). Should be the same as live operation: my scripts use systemctl to start spamd as a systemd-managed service, so spamd is running as root. ps reports the commend line as /bin/spamd -d -c -m5 -H -r /var/run/spamd.pid Unrelated to the problem at hand, but Fedora 18 and Fedora 20 does not account for as near to identical as reasonably possible. Sure, but when I set up the test environment I made sure that spamd was always started the same way as it would be in a live environment: the only difference is that its only running when I have testing to do. In any case this is a bit of a blind alley since the testing system was working just as I expected it to yesterday morning and then, after the yum upgrade the unmodified test system no longer worked that way. LATER: This morning I reran some failing examples after rebooting the test machine. No change, so I tried a few stripped-down runs, i.e. I started spamd via a test script that uses systemctl to start or stop it and then used spamc testmessage to exercise it while I played round with spamc options. In the course of this I noticed a strange effect: The FIRST message after restarting spamd never has X-Spam headers, but the second and subsequent ones do have X-Spam headers. I'm running these versions: $ spamd --version SpamAssassin Server version 3.3.2 running on Perl 5.18.2 with SSL support (IO::Socket::SSL 1.955) with zlib support (Compress::Zlib 2.062) $ spamc --version SpamAssassin Client version 3.2.4 Should the spamc/spamd version mismatch have any bad effects? The only SA package I have installed is spamassassin.i686 3.3.2-56.fc20 which came from atrpms. This is odd, since all the uninstalled packages (spamass-milter, spamass-milter-postfix, spamassassin-FuzzyOcr, spamassassin-iXhash2, spambayes, spampd, spamprobe.i686) are from Fedore repositories as you'd expect. Removing and reinstalling the spamassassin package has had no effect. I'll take this up with RedHat next week and see if I can find out why they no longer provide the main spamassassin package in their repository. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On 05/24/2014 12:23 AM, Martin Gregorie wrote: This afternoon I ran into two oddities. I haven't noticed the first in the past and have never seen the second before. 1) Missing DCC_CHECK rule = I was doing some cleanup on my private rule collection, which meant running SA 3.3.2 with the command: $ spamassassin -D data/sale01.txt 21|less so I could look at the debug output when I noticed the line: meta test DIGEST_MULTIPLE has undefined dependency 'DCC_CHECK' I don't think it has any effect on SA's operation. What is missing to cause the undefined dependency? Do I need to do anything about it. install and enable DCC in the .pre file or ignore. 2 Failure of spamc/spamd to output any X-Spam headers = This morning SA 3.3.2 was working as expected on my SA test box when I amended a rule to recognise a new spam variant. The test box is running a fully patched (as of last Friday) copy of Fedora 20. Then I did my normal weekly yum upgrade. Shortly after that I got some new spam which I ran a test on using my normal spamc/spamd test system on the SA test box. To my surprise, no X-Spam headers at all were added to it. Some experimentation showed that the headers are added if spamassassin is fed the message as I showed above (problem 1 was spotted when I was trying to work out what was losing the X-Spam headers), but if the message is passed to spamd via spamc the X-Spam headers are omitted from the output. I tried two known pieces of spam: both are small (2813 and 6984 bytes) and both showed the same effect, so it should not have been due to skipping a huge message. The yum upgrade replaced three Perl libraries: perl-IO-Socket-SSL-1.955-2.fc20.noarch perl-Image-ExifTool-9.60-1.fc20.noarch perl-Net-DNS-0.75-1.fc20.i686 Could this have caused either or both problems? doubt it.. hardly... I didn't notice anything that looked wrong in the debug report apart from the missing DCC_CHECK. This is not affecting my live mail handling box, which is still running Fedora 18 and was not upgraded today. Depending on what you have in your local.cf X headers may or not be added.
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
--As of May 23, 2014 11:23:44 PM +0100, Martin Gregorie is alleged to have said: This morning SA 3.3.2 was working as expected on my SA test box when I amended a rule to recognise a new spam variant. The test box is running a fully patched (as of last Friday) copy of Fedora 20. Then I did my normal weekly yum upgrade. Shortly after that I got some new spam which I ran a test on using my normal spamc/spamd test system on the SA test box. To my surprise, no X-Spam headers at all were added to it. --As for the rest, it is mine. Two quick questions: Does it happen to *every* message passed to spamc, and does restarting spamd solve it? This sounds similar to the behavior I was mentioning in a post earlier, and am having trouble tracking down. Restarting mitigates in my case. Daniel T. Staal --- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. ---
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 00:34 +0200, Axb wrote: On 05/24/2014 12:23 AM, Martin Gregorie wrote: This afternoon I ran into two oddities. I haven't noticed the first in the past and have never seen the second before. 1) Missing DCC_CHECK rule = install and enable DCC in the .pre file or ignore. Thanks for the advice: I don't need DCC, so I'll simply ignore it. 2 Failure of spamc/spamd to output any X-Spam headers = Depending on what you have in your local.cf X headers may or not be added. The only stuff that I changed in local.cf was typos in local rules and CVS diff config/local.cf agreed that I didn't accidentally change anything else. In case you're wondering about that pathname, my local.cf master copy lives in a normal development directory, sa_test. I prefix and tests by running a script that copies everything in config to /etc/mail/spamassassin and then runs 'spamassassin -D lint' against it. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 00:34 +0200, Axb wrote:
On 05/24/2014 12:23 AM, Martin Gregorie wrote:
2 Failure of spamc/spamd to output any X-Spam headers
=
This morning SA 3.3.2 was working as expected on my SA test box when I
amended a rule to recognise a new spam variant. The test box is running
a fully patched (as of last Friday) copy of Fedora 20. Then I did my
normal weekly yum upgrade. Shortly after that I got some new spam which
I ran a test on using my normal spamc/spamd test system on the SA test
box. To my surprise, no X-Spam headers at all were added to it.
Some experimentation showed that the headers are added if spamassassin
is fed the message as I showed above (problem 1 was spotted when I was
trying to work out what was losing the X-Spam headers), but if the
message is passed to spamd via spamc the X-Spam headers are omitted from
the output. I tried two known pieces of spam: both are small (2813 and
6984 bytes) and both showed the same effect, so it should not have been
due to skipping a huge message.
The yum upgrade replaced three Perl libraries:
Is that everything that was upgraded, or just the Perl bits?
Depending on what you have in your local.cf X headers may or not be added.
Nope, the X-Spam-Checker-Version header cannot be removed. Unless spamc
options suppress output of the message entirely, there will always be at
least that one header.
Martin, is spamd running and responding? Try pinging it:
spamc -K
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Fri, 2014-05-23 at 18:37 -0400, Daniel Staal wrote: Two quick questions: Does it happen to *every* message passed to spamc, and does restarting spamd solve it? It seems to. At least its consistently done that to a semi-random selection of my example spam collection over several tests. Each spam example is simply a text file which Evolution has dumped a spam message into under the impression that its saving it in mbox format and that has then been passed through an awk script that strips out any X-Spam headers. This sounds similar to the behavior I was mentioning in a post earlier, and am having trouble tracking down. Restarting mitigates in my case. Restarting doesn't help: I use a script to manage SA testing which: - starts spamd - starts a loop to handle every spam example named on the command line - each spam example is sent to spamd via spamc and the result passed through an awk script that only keeps X-Spam header lines. - stops spamd when the loop ends The only thing I haven't done is restart the box since the upgrade because its been busy with backups which it started before its upgrade. I'll do that ASAP when the backups end. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 01:12 +0200, Karsten Bräckelmann wrote: On Sat, 2014-05-24 at 00:34 +0200, Axb wrote: On 05/24/2014 12:23 AM, Martin Gregorie wrote: 2 Failure of spamc/spamd to output any X-Spam headers = This morning SA 3.3.2 was working as expected on my SA test box when I amended a rule to recognise a new spam variant. The test box is running a fully patched (as of last Friday) copy of Fedora 20. Then I did my normal weekly yum upgrade. Shortly after that I got some new spam which I ran a test on using my normal spamc/spamd test system on the SA test box. To my surprise, no X-Spam headers at all were added to it. Some experimentation showed that the headers are added if spamassassin is fed the message as I showed above (problem 1 was spotted when I was trying to work out what was losing the X-Spam headers), but if the message is passed to spamd via spamc the X-Spam headers are omitted from the output. I tried two known pieces of spam: both are small (2813 and 6984 bytes) and both showed the same effect, so it should not have been due to skipping a huge message. The yum upgrade replaced three Perl libraries: Is that everything that was upgraded, or just the Perl bits? Just the Perl bits. Depending on what you have in your local.cf X headers may or not be added. Nope, the X-Spam-Checker-Version header cannot be removed. Unless spamc options suppress output of the message entirely, there will always be at least that one header. Martin, is spamd running and responding? Try pinging it: spamc -K Yes, when I started it. 'spamc -K' said SPAMD/1.5 0 As you may have seen, my SA rules development test rig, which is what is doing this odd stuff, starts spamd, runs one or more spams through spamc/spamd and stops spamd. I haven't touched that script for a year or three, i.e. not since Fedora's daemon management service switched from Sys V init to systemd. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 01:10 +0100, Martin Gregorie wrote:
On Sat, 2014-05-24 at 01:12 +0200, Karsten Bräckelmann wrote:
The yum upgrade replaced three Perl libraries:
Is that everything that was upgraded, or just the Perl bits?
Just the Perl bits.
Figured as much. That rather rhetorical question was meant as a gentle
hint for you to dive down other upgrades, possibly listing them here.
Something changed, something broke. Coincidence? ;)
Martin, is spamd running and responding? Try pinging it:
spamc -K
Yes, when I started it. 'spamc -K' said SPAMD/1.5 0
By started it, do you mean starting the daemon simply for that test,
or did you ping spamd after launching your dev test rig?
Clean room vs. production conditions...
As you may have seen, my SA rules development test rig, which is what is
doing this odd stuff, starts spamd, runs one or more spams through
spamc/spamd and stops spamd. I haven't touched that script for a year or
three, i.e. not since Fedora's daemon management service switched from
Sys V init to systemd.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 02:36 +0200, Karsten Bräckelmann wrote: On Sat, 2014-05-24 at 01:10 +0100, Martin Gregorie wrote: On Sat, 2014-05-24 at 01:12 +0200, Karsten Bräckelmann wrote: The yum upgrade replaced three Perl libraries: Is that everything that was upgraded, or just the Perl bits? Just the Perl bits. Figured as much. That rather rhetorical question was meant as a gentle hint for you to dive down other upgrades, possibly listing them here. Something changed, something broke. Coincidence? ;) I've looked down the list a couple of times and didn't see anything I thought would affect it due to a possibly bad assumption that this sort of error would be insulated from the underlying binary code by at least one layer of Perl library code. I'll post the complete list either later today or on Tuesday (this is the start of the second May Bank Holiday weekend). Martin, is spamd running and responding? Try pinging it: spamc -K Yes, when I started it. 'spamc -K' said SPAMD/1.5 0 By started it, do you mean starting the daemon simply for that test, or did you ping spamd after launching your dev test rig? Yes, I meant starting it. As I said, this problem surfaced on the box I use for SA rules development, actually a Lenovo laptop. Consequently SA only runs on it during a rules test: all my test scripts are written to start spamd, run the test and stop it. Clean room vs. production conditions... Yes, exactly. I have another script that copies the SA configuration from the testing box to the live machine and then restarts the live copy of spamd. That said, the run-time environment of spamd on the testing machine is as near to identical with the live system as I can reasonably make it. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 03:28 +0100, Martin Gregorie wrote: I'll post the complete list either later today or on Tuesday (this is the start of the second May Bank Holiday weekend). Here you go. This is the yum upgrade summary: Packages Installed: kernel-PAE-3.14.4-200.fc20.i686 GraphicsMagick-doc-1.3.19-6.fc20.noarch kernel-PAE-devel-3.14.4-200.fc20.i686 GraphicsMagick-1.3.19-6.fc20.i686 Packages Updated: xorg-x11-server-common-1.14.4-9.fc20.i686 tzdata-java-2014c-1.fc20.noarch easytag-2.2.2-1.fc20.i686 xorg-x11-drv-evdev-2.8.4-1.fc20.i686 2:irqbalance-1.0.7-2.fc20.i686 systemtap-sdt-devel-2.5-2.fc20.i686 perl-Net-DNS-0.75-1.fc20.i686 xorg-x11-drv-synaptics-1.7.6-2.fc20.i686 mesa-filesystem-10.1.3-1.20140509.fc20.i686 librepo-1.7.3-1.fc20.i686 kernel-headers-3.14.4-200.fc20.i686 cifs-utils-6.3-2.fc20.i686 os-prober-1.58-6.fc20.i686 openssh-server-6.4p1-4.fc20.i686 adwaita-gtk2-theme-3.10.0-2.fc20.i686 curl-7.32.0-10.fc20.i686 mesa-libGL-10.1.3-1.20140509.fc20.i686 mesa-libxatracker-10.1.3-1.20140509.fc20.i686 openssh-6.4p1-4.fc20.i686 cups-filters-libs-1.0.53-2.fc20.i686 xorg-x11-proto-devel-7.7-10.fc20.noarch nautilus-extensions-3.10.1-4.fc20.i686 vte3-0.34.9-3.fc20.i686 mesa-libGL-devel-10.1.3-1.20140509.fc20.i686 perl-Image-ExifTool-9.60-1.fc20.noarch transmission-gtk-2.82-3.fc20.i686 transmission-2.82-3.fc20.i686 gawk-4.1.0-3.fc20.i686 selinux-policy-3.12.1-163.fc20.noarch gnome-themes-standard-3.10.0-2.fc20.i686 tzdata-2014c-1.fc20.noarch entangle-0.6.0-1.fc20.i686 mesa-libEGL-10.1.3-1.20140509.fc20.i686 gtk3-immodule-xim-3.10.9-1.fc20.i686 gtk3-3.10.9-1.fc20.i686 gdb-7.7.1-13.fc20.i686 systemtap-client-2.5-2.fc20.i686 adwaita-cursor-theme-3.10.0-2.fc20.noarch cups-filters-1.0.53-2.fc20.i686 systemtap-2.5-2.fc20.i686 libssh2-1.4.3-9.fc20.i686 firefox-29.0.1-1.fc20.i686 libcurl-7.32.0-10.fc20.i686 python-librepo-1.7.3-1.fc20.i686 openssh-clients-6.4p1-4.fc20.i686 systemtap-runtime-2.5-2.fc20.i686 python-six-1.6.1-1.fc20.noarch systemtap-devel-2.5-2.fc20.i686 mesa-dri-drivers-10.1.3-1.20140509.fc20.i686 mesa-libgbm-10.1.3-1.20140509.fc20.i686 perl-IO-Socket-SSL-1.955-2.fc20.noarch adwaita-gtk3-theme-3.10.0-2.fc20.i686 nautilus-3.10.1-4.fc20.i686 transmission-common-2.82-3.fc20.i686 libgexiv2-python2-0.10.1-1.fc20.i686 xorg-x11-server-Xorg-1.14.4-9.fc20.i686 2:qemu-guest-agent-1.6.2-5.fc20.i686 openssh-askpass-6.4p1-4.fc20.i686 transmission-cli-2.82-3.fc20.i686 libgexiv2-0.10.1-1.fc20.i686 selinux-policy-targeted-3.12.1-163.fc20.noarch mesa-libglapi-10.1.3-1.20140509.fc20.i686 To be continued next week. Thanks for all your help thus far. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-24 at 03:28 +0100, Martin Gregorie wrote:
On Sat, 2014-05-24 at 02:36 +0200, Karsten Bräckelmann wrote:
I've looked down the list a couple of times and didn't see anything I
thought would affect it due to a possibly bad assumption that this sort
of error would be insulated from the underlying binary code by at least
one layer of Perl library code.
Plain spamassassin script works, spamc/d combination does not. Nope,
this is not necessarily related to any Perl code.
Looking through the full list of upgraded packages, the two SELinux
related ones stick out. In particular, since the client / server
environment is affected, but not the standalone variant of calling the
majority of shared SA Perl code.
By started it, do you mean starting the daemon simply for that test,
or did you ping spamd after launching your dev test rig?
Yes, I meant starting it. As I said, this problem surfaced on the box I
use for SA rules development, actually a Lenovo laptop. Consequently SA
only runs on it during a rules test: all my test scripts are written to
start spamd, run the test and stop it.
Clean room vs. production conditions...
Yes, exactly.
Nah. I was talking about the difference in your test (manually starting
spamd) and the environment observed to fail (your dev test rig script).
If you want to debug what might cause spamc/d to fail, you need to
reproduce the environment as close as possible. As is, I cannot even be
sure your manual spamd test and the dev environment do use the same
port, since at least the dev script doesn't sound like root privileges
to me.
I have another script that copies the SA configuration
from the testing box to the live machine and then restarts the live copy
of spamd. That said, the run-time environment of spamd on the testing
machine is as near to identical with the live system as I can reasonably
make it.
Unrelated to the problem at hand, but Fedora 18 and Fedora 20 does not
account for as near to identical as reasonably possible.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
