On 9/10/07, Paul Griffith <[EMAIL PROTECTED]> wrote: > > Greetings, > > How do you handle Spam surges/DoS attacks? We just had a Spam surge/DoS > and are looking at ways to better withstand (as best as we can) another > surge > > > Here is how we start SA: > > -c -d -r $PIDFILE -s /var/log/spamd --socketpath=$SOCKET > --max-children=150 --min-children=10 > > Our (1) mail server is configured like this: > > CentOS 4.5 > Exim 4.67 > SpamAssassin version 3.2.3 running on Perl version 5.8.8 > ClamAV 0.91.2 (saneSecurity updates) > - handles incoming/outgoing mail > - handles imap/pop/webmail request > > Intel D Cpu 3.00Ghz with 2GB of Mem > 80GB SATA root disk > 200GB SATA mail disk (softraid mirror) > 2xIntel e1000 > > Our mail server was taking a pounding on Friday, > > Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBB > Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBB > Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBBB > ..snip... > ..snip... > ..snip... > Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states: > BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB > Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states: > BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBB > Fri Sep 7 16:17:19 2007 [26914] info: prefork: child states: > BBBBBBBBBBBBBBBBBBBBBBBBBIBBBBBBBBBBBBBBBBBBBBBBIBBB > ..snip.. > ..snip.. > Fri Sep 7 16:19:22 2007 [26914] info: prefork: child states: > > BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBSBBSBB > Fri Sep 7 16:19:23 2007 [26914] info: prefork: child states: > > BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBISBB > > At the mist of the surge we had 95 child processess running, all busy! > > Here are the sar memory stats... > > kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree > kbswpused %swpused kbswpcad > 16:10:02 16804 2056424 99.19 2900 1310880 > 2040036 208 0.01 0 > 16:20:10 37676 2035552 98.18 1872 237376 1736152 > 304092 14.90 78992 > 16:30:51 13924 2059304 99.33 1292 308944 1044160 > 996084 48.82 357444 > 16:40:02 76652 1996576 96.30 8208 1280796 1756236 > 284008 13.92 178696 > Average: 26403 2046825 98.73 5880 1364057 > 2024199 16045 0.79 6152 > > > Here are the warnings we saw in the spamd log... > > Fri Sep 7 16:20:39 2007 [26914] info: prefork: child states: > > IBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB > Fri Sep 7 16:20:40 2007 [25431] warn: spf: lookup failed: Can't locate > object method "new" via package "Net::DNS::RR::TXT" at > /xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/ > DNS/RR.pm line 312. > Fri Sep 7 16:20:41 2007 [25428] warn: spf: lookup failed: Can't locate > object method "new" via package "Net::DNS::RR::TXT" at > /xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/ > DNS/RR.pm line 312. > > Fri Sep 7 16:22:18 2007 [24684] warn: plugin: eval failed: child > processing timeout at /xsys/sbin//spamd line 1246, <GEN683> line 3398. > Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in > pattern match (m//) at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 211, > <GEN749> line 3398. > Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in > scalar assignment at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 227, > <GEN749> line 3398. > > Fri Sep 7 16:26:15 2007 [25227] info: spamd: clean message (1.5/5.0) for > cs242027:9190 in 406.1 seconds, 243776 bytes. > Fri Sep 7 16:26:19 2007 [24688] warn: spamd: copy_config timeout, > respawning child process after 1 messages at /xsys/sbin//spamd line 1103. > Fri Sep 7 16:26:24 2007 [25046] warn: spamd: copy_config timeout, > respawning child process after 1 messages at /xsys/sbin//spamd line 1103. > Fri Sep 7 16:26:28 2007 [24692] warn: spamd: copy_config timeout, > respawning child process after 1 messages at /xsys/sbin//spamd line 1103. > > Fri Sep 7 16:30:35 2007 [26914] info: spamd: server successfully spawned > child process, pid 26312 > Fri Sep 7 16:30:37 2007 [24685] warn: spamd: copy_config timeout, > respawning child process after 1 messages at /xsys/sbin//spamd line 1103. > Fri Sep 7 16:30:39 2007 [26914] warn: prefork: cannot ping 24702, file > handle not defined, child likely to still be processing SIGCHLD handler > after killing itself > Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killing failed child 24702 > fd=undefined at > /xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line > 171. > Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killed child 24702 > Fri Sep 7 16:30:41 2007 [26914] info: spamd: handled cleanup of child pid > 24702 due to SIGCHLD > Fri Sep 7 16:30:41 2007 [26914] warn: prefork: cannot ping 24687, file > handle not defined, child likely to still be processing SIGCHLD handler > after killing itself > Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killing failed child 24687 > fd=undefined at > /xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line > 171. > Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killed child 24687 > > > Looking at the swap usage, I was thinking I would be better if I reduced > the number of children processes and let thing queue up. I know I will > also have to look at exim and it's ratelimit command. Any other idea's on > handling spam surges/DoS? > > Thanks > Paul >
At my site we operate under the presumption that SpamAssassin should be avoided if at all possible because it is so expensive on our resources compared to some other easy checks. This helps us to deal with DoS and "surges" from retarded bots quite well (so far at least). We reduce the messages bound for SA to less than 10% of our traffic by a combination of postfix UCE checks, a couple very accurate RBLs, selective greylisting and our own whitelist. When the surges/DOS happen, they tend to increase the number of messages thrown away but rarely effect the volume running through SA. -Aaron
