Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sun, 2014-06-01 at 01:20 +0100, Martin Gregorie wrote:
> On Sun, 2014-06-01 at 01:21 +0200, Karsten Bräckelmann wrote:
> > For bonus-points, watch the logs for spamd claiming to be ready.
>
> Here you go:
> Jun 1 01:07:41 zappa spamd[15831]: plugin: eval failed: Insecure
> dependency in connect while running with -T switch
> at /usr/lib/perl5/IO/Socket.pm line 115.
> Jun 1 01:07:42 zappa spamd[15832]: spamd: connection from
> zappa.gregorie.lan [127.0.0.1] at port 39842
> Jun 1 01:07:42 zappa spamd[15832]: spamd: setuid to kiwi succeeded
> Jun 1 01:07:42 zappa spamd[15832]: spamd: Insecure dependency in
> connect while running setuid at /usr/lib/perl5/IO/Socket.pm line 115,
> line 9.
> Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: II
> Jun 1 01:07:44 zappa spamd[15832]: spamd: connection from
> zappa.gregorie.lan [127.0.0.1] at port 39843
> Jun 1 01:07:44 zappa spamd[15832]: spamd: setuid to kiwi succeeded
> Jun 1 01:07:44 zappa spamd[15832]: spamd: processing message (unknown)
> for kiwi:1000
> I think the 2nd 'insecure dependency' at 01:07:42 is associated with
> spamd receiving the first message
If so, that would be the one passed back unprocessed, while the
following invocation works.
That error message rings a bell. Will (aragonx?) posted that line very
recently, and updated the thread himself just today, pointing to a RH /
Fedora 20 bugzilla report for its SA 3.3.2 package, related to Perl
Net::DNS 0.76 (once available).
https://bugzilla.redhat.com/show_bug.cgi?id=1096405
Comment 5 also mentions an issue with Perl Net::DNS 0.75, which is the
exact version the package upgrade pulled in on your system before this
started. The report matches the first "insecure dependency" above.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sun, 2014-06-01 at 01:21 +0200, Karsten Bräckelmann wrote: > I haven't really used systemd yet, but one fundamental design decision > is, that systemd itself takes care about sockets and stuff, returning > early and asynchronously lets the service complete starting up in the > background. > That could easily fit this scenario. > This might explain what you are observing, the first spamc call failing, > if you do run spamc close after spamd service starting. Can you still > reproduce the issue, if you wait a minute before running spamc? > > Did you ever verify spamd was running, or did you just run your usual > test script? > Yep. Thats implied with a separate spamd start: sometimes I check that its running (script wrapper round 'systemctl status spamassassin') but you can tell from inspection because the testsa script takes a lot longer to run if it has to start spamd. > Also, IIRC you never mentioned the spamc exit code. See the man-page for > details, spamc differentiates between a lot of network and other error > conditions. > =>Here's the instant 'start spamd, run spamc, stop spamd test: $ sudo systemctl start spamassassin; echo $?; echo | spamc; echo $?; sudo systemctl stop spamassassin 0 0 $ ===>and here's the version with a delay after starting: $ sudo systemctl start spamassassin; echo $?; sleep 30 0 $ echo | spamc; echo $? 0 $ sudo systemctl stop spamassassin $ =>same again but with 2 messages and no delay $ sudo systemctl start spamassassin; echo $?; spamc as the last, but with a 2 sec delay between messages $ sudo systemctl start spamassassin; echo $?; spamc For bonus-points, watch the logs for spamd claiming to be ready. > > Here you go: Jun 1 01:07:41 zappa spamd[15831]: plugin: eval failed: Insecure dependency in connect while running with -T switch at /usr/lib/perl5/IO/Socket.pm line 115. Jun 1 01:07:42 zappa spamd[15831]: spamd: server started on port 783/tcp (running version 3.3.2) Jun 1 01:07:42 zappa spamd[15831]: spamd: server pid: 15831 Jun 1 01:07:42 zappa spamd[15831]: spamd: server successfully spawned child process, pid 15832 Jun 1 01:07:42 zappa spamd[15831]: spamd: server successfully spawned child process, pid 15833 Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: IS Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: II Jun 1 01:07:42 zappa spamd[15832]: spamd: connection from zappa.gregorie.lan [127.0.0.1] at port 39842 Jun 1 01:07:42 zappa spamd[15832]: spamd: setuid to kiwi succeeded Jun 1 01:07:42 zappa spamd[15832]: spamd: Insecure dependency in connect while running setuid at /usr/lib/perl5/IO/Socket.pm line 115, line 9. Jun 1 01:07:42 zappa spamd[15831]: prefork: child states: II Jun 1 01:07:44 zappa spamd[15832]: spamd: connection from zappa.gregorie.lan [127.0.0.1] at port 39843 Jun 1 01:07:44 zappa spamd[15832]: spamd: setuid to kiwi succeeded Jun 1 01:07:44 zappa spamd[15832]: spamd: processing message (unknown) for kiwi:1000 Jun 1 01:07:44 zappa spamd[15832]: spamd: clean message (5.6/6.0) for kiwi:1000 in 0.1 seconds, 31 bytes. Jun 1 01:07:44 zappa spamd[15832]: spamd: result: . 5 - MG_WRONG_DOMAIN,MISSING_DATE,MISSING_FROM,MISSING_MID,MISSING_SUBJECT,NO_RECEIVED,NO_RELAYS,TO_MALFORMED scantime=0.1,size=31,user=kiwi,uid=1000,required_score=6.0,rhost=zappa.gregorie.lan,raddr=127.0.0.1,rport=39843,mid=(unknown),autolearn=no Jun 1 01:07:44 zappa spamd[15831]: spamd: server killed by SIGTERM, shutting down I think the 2nd 'insecure dependency' at 01:07:42 is associated with spamd receiving the first message Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-31 at 23:07 +0100, Martin Gregorie wrote:
> On Sat, 2014-05-31 at 20:15 +0200, Karsten Bräckelmann wrote:
> > > The testsa script looks like this:
> >
> > > state=$(spamdstatus)
> > > if [ "$state" == 'spamd is stopped' ]
> > > then
> > > sudo systemctl start spamassassin.service
> > > fi
> >
> > Most likely unrelated, though this code might potentially mess with
> > spamd unnoticed. I'd get rid of that part while debugging the issue, or
> > at least make it clearly print a warning.
>
> This script looks like this:
> So I don't see how it can mess with spamd.
By "messing with" I mean code that potentially (re)starts spamd, without
us knowing. That is an absolute no-go when debugging or tracking down
bugs.
> In any case the first time
> failure happens regardless of whether testsa is allowed to start and
> stop spamd of it it is started separately. The only difference is that
> if testsa does the start/stop it always fails since every test it
> submits is the first spamc request after a restart, while if spamd is
> started separately, only the first test fails to scan the message: the
> rest are OK until spamd is stopped and restarted.
Actually, thinking about that (re)starting of spamd, and only the first
attempt failing:
I haven't really used systemd yet, but one fundamental design decision
is, that systemd itself takes care about sockets and stuff, returning
early and asynchronously lets the service complete starting up in the
background.
This might explain what you are observing, the first spamc call failing,
if you do run spamc close after spamd service starting. Can you still
reproduce the issue, if you wait a minute before running spamc?
Did you ever verify spamd was running, or did you just run your usual
test script?
Also, IIRC you never mentioned the spamc exit code. See the man-page for
details, spamc differentiates between a lot of network and other error
conditions.
Generally, while debugging $product issues, you should first get complex
custom wrappers out of the equation. With spamc, a stripped down test
case is:
echo | spamc
echo $?
No complex wrapper, no spam needed. Generate minimal input. There will
always be at least an X-Spam-Version header, if it passed spamd. Get the
exit code in case of failure (no X-Spam header), compare with the
man-page.
Run once *immediately* after starting spamd as usual. Stop and restart
spamd as usual, wait, and try stripped down test case again.
For bonus-points, watch the logs for spamd claiming to be ready.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-31 at 20:15 +0200, Karsten Bräckelmann wrote: > $ which -a spamc > 'locate spamc' turned up a copy of spamc 3.2.4 in /usr/local/bin dated 2008. I can't remember how it might have got there since I've only ever installed SA from the Fedora repo. Anyway, that is gone now and both spamc and spamd are at version 3.3.2 The only copy of spamd is 3.3.2 and AFAICT there have never been additional copies of it or spamassassin - also 3.3.2 > Given that still unexplained version mismatch, it is not unlikely there > are bad installations of Perl libraries, too. Might be time to install > the system fresh... > That happened a few weeks back when I did a clean install of Fedora 20. This replaced all executables that aren't in /usr/local/bin: the /usr/local directory is a symlink to /home/local and home is a separate partition and is the only one on this system that isn't reformatted during a clean install. > > The testsa script looks like this: > > > state=$(spamdstatus) > > if [ "$state" == 'spamd is stopped' ] > > then > > sudo systemctl start spamassassin.service > > fi > > Most likely unrelated, though this code might potentially mess with > spamd unnoticed. I'd get rid of that part while debugging the issue, or > at least make it clearly print a warning. > This script looks like this: #!/bin/bash sudo systemctl status spamassassin.service 2>&1 >/dev/null if [ $? -eq 0 ] then reply='spamd is running' else reply='spamd is stopped' fi echo $reply So I don't see how it can mess with spamd. In any case the first time failure happens regardless of whether testsa is allowed to start and stop spamd of it it is started separately. The only difference is that if testsa does the start/stop it always fails since every test it submits is the first spamc request after a restart, while if spamd is started separately, only the first test fails to scan the message: the rest are OK until spamd is stopped and restarted. > Apart from that, the issue surfaced after a system upgrade, without any > SA code changed. Makes the distro the more likely place for reporting. > Yeah, I'll raise a bug with Fedora and spare this mailing list from more bandwidth use. Martin
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Sat, 2014-05-31 at 17:39 +0100, Martin Gregorie wrote:
> On Tue, 2014-05-27 at 02:48 +0200, Karsten Bräckelmann wrote:
> > A quick googlin' brings up spamassassin 3.3.2-18.fc20 for Fedora 20, in
> > a single package shipping both spamc and spamd in /usr/bin.
>
> After deleting and reinstalling (yum install spamassassin) I now have
> spamassassin.i686 3.3.2-18.fc20 installed but with no apparent effect:
> it is running these versions:
>
> # spamd --version
> SpamAssassin Server version 3.3.2
> running on Perl 5.18.2
> with SSL support (IO::Socket::SSL 1.955)
> with zlib support (Compress::Zlib 2.062)
> # spamc --version
> SpamAssassin Client version 3.2.4
$ which -a spamc
The Fedora spamassassin 3.3.2 RPM you just installed does ship spamc
in /usr/bin. The version mismatch persists, so there is another spamc
from a different source early in your $PATH.
Use 'rpm -qf ' to find out which package it belongs to, if any.
In either case, packaged or built from source, we're chasing something
in your environment or system which you are obviously unaware of.
Assuming you are the only one administrating that system: Without
pointing fingers, the system has to be considered dirty.
Given that still unexplained version mismatch, it is not unlikely there
are bad installations of Perl libraries, too. Might be time to install
the system fresh...
> The testsa script looks like this:
> state=$(spamdstatus)
> if [ "$state" == 'spamd is stopped' ]
> then
> sudo systemctl start spamassassin.service
> fi
Most likely unrelated, though this code might potentially mess with
spamd unnoticed. I'd get rid of that part while debugging the issue, or
at least make it clearly print a warning.
> Obviously something is wrong, but should this be raised on the Fedora
> Bugzilla or would it be better raised within the SA project?
I suggest to first try to reproduce the issue on a fresh installed
system.
Apart from that, the issue surfaced after a system upgrade, without any
SA code changed. Makes the distro the more likely place for reporting.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Unexpected missing rule name, failure of spams/spamd to output X-Spam headers
On Tue, 2014-05-27 at 02:48 +0200, Karsten Bräckelmann wrote:
> On Sat, 2014-05-24 at 12:34 +0100, Martin Gregorie wrote:
> > On Sat, 2014-05-24 at 05:04 +0200, Karsten Bräckelmann wrote:
>
> > LATER: This morning I reran some failing examples after rebooting the
> > test machine. No change, so I tried a few stripped-down runs, i.e. I
> > started spamd via a test script that uses systemctl to start or stop it
> > and then used "spamc > with spamc options. In the course of this I noticed a strange effect:
> >
> > The FIRST message after restarting spamd never has X-Spam headers, but
> > the second and subsequent ones do have X-Spam headers.
> >
> > I'm running these versions:
> >
> > $ spamd --version
> > SpamAssassin Server version 3.3.2
> > running on Perl 5.18.2
> > with SSL support (IO::Socket::SSL 1.955)
> > with zlib support (Compress::Zlib 2.062)
> > $ spamc --version
> > SpamAssassin Client version 3.2.4
> >
> > Should the spamc/spamd version mismatch have any bad effects?
>
> No, there should be absolutely no problems, since spamc/d are using a
> protocol that hasn't even added features since 3.2, let alone changed in
> an incompatible way.
>
> It does indicate a real problem with your package management or custom
> builds, though.
>
>
> > The only SA package I have installed is spamassassin.i686 3.3.2-56.fc20
> > which came from atrpms. This is odd, since all the uninstalled packages
> > (spamass-milter, spamass-milter-postfix, spamassassin-FuzzyOcr,
> > spamassassin-iXhash2, spambayes, spampd, spamprobe.i686) are from Fedore
> > repositories as you'd expect. Removing and reinstalling the spamassassin
> > package has had no effect.
> >
> > I'll take this up with RedHat next week and see if I can find out why
> > they no longer provide the main spamassassin package in their
> > repository.
>
> A quick googlin' brings up spamassassin 3.3.2-18.fc20 for Fedora 20, in
> a single package shipping both spamc and spamd in /usr/bin.
>
After deleting and reinstalling (yum install spamassassin) I now have
spamassassin.i686 3.3.2-18.fc20 installed but with no apparent effect:
it is running these versions:
# spamd --version
SpamAssassin Server version 3.3.2
running on Perl 5.18.2
with SSL support (IO::Socket::SSL 1.955)
with zlib support (Compress::Zlib 2.062)
# spamc --version
SpamAssassin Client version 3.2.4
exactly as before. Yum says it knows about these spamassassin package:
Installed Packages
spamassassin.i686 3.3.2-18.fc20 @updates
Available Packages
spamassassin.i686 3.3.2-56.fc20 atrpms
which at least explains why the atrpms package was installed rather than
the Fedora @updates one.
The only problem is as before
$ start sa# private script which uses systemctl start spamassassin
# which is running under root as expected
$ ps -ef |grep spamd
root 23426 1 6 17:28 ?00:00:03 /usr/bin/spamd
--pidfile /var/run/spamd.pid -d -c -m5 -H
root 23427 23426 0 17:28 ?00:00:00 spamd child
root 23428 23426 0 17:28 ?00:00:00 spamd child
kiwi 23449 23142 0 17:29 pts/100:00:00 grep --color=auto spamd
$ testsa data/sale57.txt
Checking data/sale57.txt
Checked 1 messages in 0 secs, 0 mS/message
$ testsa data/sale57.txt
Checking data/sale57.txt
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on
zappa.gregorie.lan
X-Spam-Flag: YES
X-Spam-Level: ***
X-Spam-Status: Yes, score=7.9 required=6.0 tests=HTML_IMAGE_RATIO_04,
HTML_MESSAGE,MG_MONEY,MG_PRICE,MG_PRODUCT,MG_SALE,MG_SALESOFFER,MG_SALESPAM,
MG_SPAMREF,RP_MATCHES_RCVD,T_DKIM_INVALID,T_END_FUTURE_EMAILS,
UNPARSEABLE_RELAY autolearn=no version=3.3.2
Checked 1 messages in 0 secs, 0 mS/message
$ sa stop
As you can see, spamc/spamd from @updates does exactly the same as the
one from atrpms: the first message passed by spamc after spamd is
started apparently does nothing but the second and subsequent ones are
scanned.
The testsa script looks like this:
$ cat testsa
#!/bin/bash
#
#help
# Syntax: testsa [-x] testmessage.
# Function: Run one or more messages through spamc
# The output is filtered to contain only headers
# added by Spamassassin
# Options: -x show the whole message
#end
if [ "$1" == '-?' ]
then
script_help testsa
exit 1
fi
state=$(spamdstatus)
if [ "$state" == 'spamd is stopped' ]
then
sudo systemctl start spamassassin.service
fi
if [ "$1" == '-x' ]
then
filter=no
shift
else
filter=yes
fi
tcount=0
starttime=$(date +%s)
for s
do
echo "Checking $s"
echo ""
if [ $filter == 'yes' ]
then
spamc --max-size=100 <$s | gawk '
BEGIN { tag=0 }
/^X-Spam/ { tag=1; print; next }
/^ / || /^\t/ { if (tag==1) { print } next }
{ tag = 0 }
Re: writing rules howto?
Andreas Schulze: Kasten, sorry -> Karsten works wonderful. I now have a list of hostnames SA find in the messagebody as new header! Thanks. Much simpler then I thought... Andreas
Re: writing rules howto?
Karsten Bräckelmann: > Since SA 3.4, there are template tags which already might be all you > need. The template tags _URIHOSTS_ and _URIDOMAINS_ list all extracted > (and to be looked up) URIs, including full hostname and domain only > respectively. No path information. > > add_header all UriHosts _URIHOSTS_ > > will add an X-Spam-UriHosts header. Since this actually is provided by > the URIDNSBL plugin, skiplist and max number apply as outlined. Kasten, thanks for these comprehensive answers. I think they are valuable pointers. Andreas
