RE: Server overload, queuing for SA possible?
Bowie Bailey wrote: There is a --no-safe-fallback option on spamc which will cause it to exit with an error message in the case of any problems (Normally, it always exits with a 0 exit status). If you don't want anything to go through unscanned, you can try this setting. If I have 5 spamd children available and (just to torture it) I fire off 50 spamc processes, what happens? Each spamc process will try to connect to spamd three times at 1 second intervals. Any processes that fail to connect will let their message through unscanned. The number of retries and the sleep interval between them can be configured on the spamc command line. Thanks for all the hints. I have now tried the following: I added these options to spamd --max-children=4 --timeout-child=100 and spamc --retry-sleep=20 --no-safe-fallback. Monky I will wait and see if this improves the situation. Adding memory is of course tempting, but the server is a rented one where I cannot change the hardware but by upgrading to another (more costly) server hosting package. -- View this message in context: http://www.nabble.com/Server-overload%2C-queuing-for-SA-possible--tp22636540p22701919.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Server overload, queuing for SA possible?
Hallo list, receiving a bunch of obvious spam emails without the SA tags in it made me look at my logfiles and I found out - thats what I guess - that for a short time my server was reaching his limits. Short grep extracts from my logfile: Mar 21 10:22:56 h1306680 spamd[9247]: prefork: child states: II Mar 21 10:54:11 h1306680 spamd[9247]: prefork: child states: II Mar 21 11:00:13 h1306680 spamd[9247]: prefork: child states: BB Mar 21 11:01:43 h1306680 spamd[9247]: prefork: child states: IBI Mar 21 11:07:40 h1306680 spamd[9247]: prefork: child states: BIB Mar 21 11:12:53 h1306680 spamd[9247]: prefork: child states: BBB Mar 21 11:13:11 h1306680 spamd[9247]: prefork: child states: Mar 21 11:14:28 h1306680 spamd[9247]: prefork: child states: I Mar 21 11:14:48 h1306680 spamd[9247]: prefork: child states: B Mar 21 11:15:31 h1306680 spamd[9247]: prefork: server reached --max-children setting, consider raising it Mar 21 11:16:01 h1306680 spamd[9247]: prefork: child states: B Mar 21 11:16:10 h1306680 spamd[9247]: prefork: server reached --max-children setting, consider raising it Mar 21 11:16:22 h1306680 spamd[9247]: prefork: child states: B Mar 21 11:17:16 h1306680 spamd[9247]: prefork: server reached --max-children setting, consider raising it Mar 21 11:17:38 h1306680 spamd[9247]: prefork: child states: B Mar 21 11:17:38 h1306680 spamd[9247]: prefork: server reached --max-children setting, consider raising it Mar 21 11:17:38 h1306680 spamd[9247]: prefork: child states: B Mar 21 11:17:55 h1306680 spamd[9247]: prefork: server reached --max-children setting, consider raising it Mar 21 11:17:55 h1306680 spamd[9247]: prefork: child states: I Mar 21 11:17:55 h1306680 spamd[9247]: prefork: child states: IBBBI Mar 21 11:18:09 h1306680 spamd[9247]: prefork: child states: IBBII Mar 21 11:18:09 h1306680 spamd[9247]: prefork: child states: IBBIK Mar 21 11:18:13 h1306680 spamd[9247]: prefork: child states: IIBI Mar 21 11:18:50 h1306680 spamd[9247]: prefork: child states: IIB Mar 21 11:19:04 h1306680 spamd[9247]: prefork: child states: III Mar 21 10:22:56 h1306680 spamd[3550]: spamd: identified spam (22.1/5.0) for popuser:110 in 3.1 seconds, 1096 bytes. Mar 21 10:59:07 h1306680 spamd[9310]: spamd: identified spam (16.4/5.0) for popuser:110 in 116.4 seconds, 4195 bytes. Mar 21 10:59:08 h1306680 spamd[3550]: spamd: identified spam (16.4/5.0) for popuser:110 in 132.1 seconds, 4195 bytes. Mar 21 11:04:49 h1306680 spamd[9310]: spamd: identified spam (16.9/5.0) for popuser:110 in 275.9 seconds, 4195 bytes. Mar 21 11:11:48 h1306680 spamd[3550]: spamd: identified spam (32.3/5.0) for popuser:110 in 373.2 seconds, 17910 bytes. Mar 21 11:12:47 h1306680 spamd[30139]: spamd: identified spam (14.8/5.0) for popuser:110 in 412.3 seconds, 4164 bytes. Mar 21 11:13:13 h1306680 spamd[9310]: spamd: identified spam (15.3/5.0) for popuser:110 in 115.2 seconds, 4164 bytes. Mar 21 11:14:22 h1306680 spamd[3550]: spamd: identified spam (15.6/5.0) for popuser:110 in 88.2 seconds, 4164 bytes. Mar 21 11:16:08 h1306680 spamd[32043]: spamd: identified spam (20.3/5.0) for popuser:110 in 176.5 seconds, 3958 bytes. Mar 21 11:16:22 h1306680 spamd[30139]: spamd: identified spam (16.8/5.0) for popuser:110 in 114.4 seconds, 3980 bytes. Mar 21 11:16:39 h1306680 spamd[9310]: spamd: identified spam (20.3/5.0) for popuser:110 in 111.8 seconds, 3958 bytes. Mar 21 11:17:00 h1306680 spamd[32266]: spamd: identified spam (20.3/5.0) for popuser:110 in 131.9 seconds, 3958 bytes. Mar 21 11:17:44 h1306680 spamd[3550]: spamd: identified spam (16.8/5.0) for popuser:110 in 102.5 seconds, 3980 bytes. Mar 21 11:18:01 h1306680 spamd[32043]: spamd: identified spam (16.8/5.0) for popuser:110 in 98.1 seconds, 3980 bytes. Mar 21 11:18:13 h1306680 spamd[9310]: spamd: identified spam (27.8/5.0) for popuser:110 in 34.5 seconds, 3166 bytes. Mar 21 11:18:32 h1306680 spamd[30139]: spamd: identified spam (24.0/5.0) for popuser:110 in 53.9 seconds, 1292 bytes. Mar 21 11:19:33 h1306680 spamd[3550]: spamd: identified spam (32.9/5.0) for popuser:110 in 17.7 seconds, 992 bytes. Mar 21 11:19:50 h1306680 spamd[3550]: spamd: identified spam (26.5/5.0) for popuser:110 in 10.2 seconds, 1244 bytes. Mar 21 11:20:33 h1306680 spamd[9310]: spamd: identified spam (18.1/5.0) for popuser:110 in 23.9 seconds, 1238 bytes. What I make of this is that when my server is using his maximum of 5 spamd children he hits the RAM limit and starts paging (the explosion of scanning time). Is this a sensible assessment? What could I do about it? Raising --maximum-children seems not a good idea. Actually it seems wiser to reduce to a maximum of 4 children. How can I prevent spam from passing my system unchecked due to a (temporary) overload? If I look at the prefork child states the critical time is followed by hours of II / BI. How could I queue the incoming emails and ensure that every email gets scanned? Currently I am using qmail's defaultdelivery setting: | spamc |
Re: Some emails pass spamassassin unprocessed
Karsten Bräckelmann-2 wrote: Mentioning some numbers is good, though too qualitative. How many mails is that per day, in absolute numbers? Maybe 1-10 a day in total. I have several email accounts there and it happens with all of them although not in a serious amount per account. Karsten Bräckelmann-2 wrote: Of course, never forget that there's a default max-size per message. Unless told otherwise, spamc won't scan mail that's larger than 500 kByte. Probably not your issue, though, unless (most) of the unprocessed mail you're talking about actually *is* large. The problem is not size related. The spam emails that pass often are only one line text mails. My settings do not specify a max-size since the default value seems to suit me. Karsten Bräckelmann-2 wrote: Also, this is the spamc default of safe fallback. That means, if there is any issue in communicating with the daemon, the message will be passed along unprocessed. But let's re-schedule that for later. (See follow-up post.) The logfiles do not report any problems. I ran procmail with VERBOSE=yes for some time now and it shows that even the unprocessed emails get passed to spamc. Karsten Bräckelmann-2 wrote: Unrelated: Are you using mbox or maildir storage? The $MAILDIR hints you actually are using maildir format, though the junk folder is an mbox file! With mbox, you seriously should add locking to any delivering recipe. Maildir is beeing used as maildir storage. I just use the mbox file for detected spam emails instead of forgetting them right away. Thanks for the hint with locking, I added the colon to the line concerning the mbox write access. Karsten Bräckelmann-2 wrote: Yes. Check your logs. If you are running out of spamd children, you'll see something like this in the logs: prefork: child states One state indicating char per children. B is busy, I means idle. Idle processes are ready to take a message. If you're seeing too many busy children, you're server can't handle the load. In that case, you /can/ increase the number of children, if you got plenty of RAM. Limiting the parallel resource usage by locking (in procmail) can help, too. And it definitely would be worth investigating more, like how long the children take for processing a message. Too long scan times can amplify this. guenther I checked the logs and that does not seem to be the cause of the problem. My log shows 'II' in most cases and rarely more than 3 threads running. My max threads is 5 I think. I try and add locking to the spamc call now in all the .procmailrc files. I will come back to you =) Many thanks for all the hints so far! -- View this message in context: http://www.nabble.com/Some-emails-pass-spamassassin-unprocessed-tp22119041p22355105.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Some emails pass spamassassin unprocessed
Hello, I am running the Spamd Daemon version 3.2.5 on my Linux web and mail server and in general it works well. From time to time (somewhere in between 1-10% of all emails) spam passes the filter - but not because spamassassin decides that it is ham but because the email never gets processed by spamassassin (the header shows no X-Spam at all). This is how I pass the emails to spamassassin: I am running qmail and each user has a identical .qmail file: # more .qmail |preline /usr/bin/procmail -m .procmailrc The .procmailrc looks like this: # more .procmailrc LOGFILE=procmail.log HOME=[... this folder] MAILDIR=$HOME/Maildir :0fw |spamc :0 * ^X-Spam-Flag: YES $HOME/junk :0 w | /usr/bin/deliverquota $MAILDIR As I mentioned before everything works fine in 90+% of all incoming mails. Any hints on what might happen to the emails that simply get passed along unchecked? Could it be that if the server is busy processing other email he skips a few? Any hints are welcome, thanks in advance. -- View this message in context: http://www.nabble.com/Some-emails-pass-spamassassin-unprocessed-tp22119041p22119041.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: spamassassin versions
Karsten Bräckelmann-2 wrote: How did you install SA, both the previous and the latest version? With a source or CPAN install, it is likely you didn't uninstall the previous version properly, leaving behind some stale files. Also, different prefixes used might be the culprit here. With a package install (RPM or DEB) my guess is, that your distro / packager split SA in multiple packages, and you just forgot to update the spamd package. guenther Thanks for your hints. The recent version (3.2.5) I installed from CPAN and you were right, only Mail:SpamAssassin was installed, not Mail:SpamAssassin:Spamd. I installed spamd from CPAN now but it did not overwrite the old version. The old version was installed from a package as far as I remember. Is there a generic way to uninstall the old spamd version? Sorry for this general question, my linux knowledge is very limited. -- View this message in context: http://www.nabble.com/spamassassin-versions-tp22062607p22079428.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: spamassassin versions
Thank you all very much. I think I managed to get rid of the old version and have a complete new version up and running. Your hints helped me remember and figure out how to do it. I uninstalled the old version with YAST and installed the new from CPAN (only Spamd needed). I just needed to make sure the config files were used and it seems to work again now. -- View this message in context: http://www.nabble.com/spamassassin-versions-tp22062607p22092905.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
spamassassin versions
Hallo, I have installed spamassassin 3.2.5 on my linux server but when I check the version I get those messages: h1306680:/usr/bin # spamd --version SpamAssassin Server version 3.1.8 running on Perl 5.8.8 with SSL support (IO::Socket::SSL 0.97) h1306680:/usr/bin # spamc --version SpamAssassin Client version 3.2.5 h1306680:/usr/bin # spamassassin --version SpamAssassin version 3.2.5 running on Perl version 5.8.8 Did something go wrong with the installation or has the spamd part not been updated in the recent versions? Thanks for any hints. -- View this message in context: http://www.nabble.com/spamassassin-versions-tp22062607p22062607.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
