[Johannes, I hope it's ok to cc you this as your contact instructions say to do it for dovecot-antispam emails sent to dovecot mailing list and while I send this one to dspam list I hope you still would find it relevant]
Hello,
I've been using dspam with Dovecot and its antispam plugin for quite some
time without any problems but since relatively recently reclassification of
false negatives as spams stopped working and, of course, the quality of
spam detection started to suffer. The symptom of the problem is the
following log message in syslog:
dspam[pid]: Unable to attach DSPAM context
dspam[pid]: process_message returned error -2. dropping message.
As there is no information about what exactly went wrong there I rebuilt
dspam with debug info and found that the real cause of the problem was that
mmap() in _hash_drv_open() was failing with ENOMEM ("Cannot allocate
memory") which explains why does it fail.
Unfortunately why does mmap() itself fail is a complete mystery to me.
Here are some of my findings:
1. There is plenty (in excess of a couple gigabytes, compared to ~250MB
size of the .css file being mapped) of available physical and virtual
memory on this Linux amd64 system (see below for full details).
2. A simple test program calling mmap() on the same .css file works (its
full source is included at the end of the message).
3. Even more surprisingly, dspam itself works fine when called from command
line as "dspam --source=error --class=spam --signature=... --user zeitlin"
4. Unfortunately the test program still works even when called from antispam
plugin so it's not Dovecot/antispam "fault" neither.
5. As a final data point, cssstats dspam tool which also uses the same
_hash_drv_open() succeeds -- when called from command line -- too.
I have absolutely no ideas about what could explain this behaviour. FWIW
I'm not the only one seeing it, see e.g.
http://www.shapeshifter.se/2009/03/03/dspam-converting-from-the-hash-driver-to-mysql-driver/
which speaks about exactly the same error in a very similar configuration
and shows that it's not specific to Linux as this post is about FreeBSD.
And there are plenty of other reports of "Unable to attach DSPAM context"
which might be due to this problem (but it could also be due to permission
problems of course...).
The trouble is that nobody, including the author of the blog post above,
found the real reason for mmap() failure. Personally I thought that, as
mmap() mappings are inherited by the child process, maybe dovecot or
antispam plugin might be mmap()-ing something huge into the process address
space and not munmap()-ing it before spawning dspam. But there is nothing
like this in /proc/<imap-pid>/maps and I even tried setting "mmap_disable =
yes" in /etc/dovecot/dovecot.conf without any effect (well, I did see that
maps for the .index file disappeared from /proc/pid/maps, so it was taken
into account, it just didn't change anything for the mmap() call failure in
dspam). And I can't think of any other reason for mmap() to fail with
ENOMEM.
For the reference, here are the full details of my system:
- OS: Debian GNU/Linux 4.0 on x86_64
- Kernel: 2.6.20
- DSPAM: latest from git
- Dovecot: 1.0.13-1~bpo40+1 (Debian backports repository)
- Dovecot-antispam from git (behaviour was the same with 1.0 I used before)
Dovecot configuration is pretty standard, here is the output of "dovecot -n":
# 1.0.13: /etc/dovecot/dovecot.conf
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap
login_dir: /var/run/dovecot/login
login_executable: /usr/lib/dovecot/imap-login
login_greeting_capability: yes
mail_privileged_group: mail
mail_location: mbox:%h/mail:INBOX=/var/mail/%u
mbox_lazy_writes: no
mail_plugins: antispam
auth default:
passdb:
driver: pam
userdb:
driver: passwd
plugin:
antispam_signature: X-DSPAM-Signature
antispam_spam: Junk
antispam_dspam_binary: /usr/local/bin/dspam
antispam_dspam_args: --user;%n
Does anybody have any ideas about what is going on here? Thanks in advance
for any ideas!
VZ
Appendix:
Source of the test program used to test mmap() which can be executed from
dovecot-antispam plugin by setting antispam_dspam_binary to its path and
antispam_dspam_args to the name of the .css file to map:
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <syslog.h>
#include <sys/mman.h>
#include <sys/types.h>
static const int logprio = LOG_LOCAL0 | LOG_DEBUG;
int main(int argc, char **argv)
{
int fd;
off_t len;
const char *fname;
if ( argc == 1 ) {
/* ignore any preceding arguments to allow calling it as dspam */
syslog(logprio, "Usage: %s [...whatever...] <file>", argv[0]);
return 2;
}
fname = argv[argc - 1];
fd = open(fname, O_RDWR);
if ( fd == -1 ) {
syslog(logprio, "open(\"%s\") failed: %s", fname, strerror(errno));
return 1;
}
len = lseek(fd, 0, SEEK_END);
if ( len == -1 ) {
syslog(logprio, "lseek(SEEK_END) failed: %s", strerror(errno));
return 1;
}
if ( mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0) ==
MAP_FAILED) {
syslog(logprio, "mmap() failed: %s", strerror(errno));
return 1;
}
syslog(logprio, "mmap(\"%s\", 0..%ld) succeeded.\n", fname, len);
return 0;
}
pgpJdR9eWTM4F.pgp
Description: PGP signature
------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________ Dspam-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspam-devel
