Greetings,
How do you handle Spam surges/DoS attacks? We just had a Spam surge/DoS
and are looking at ways to better withstand (as best as we can) another
surge
Here is how we start SA:
-c -d -r $PIDFILE -s /var/log/spamd --socketpath=$SOCKET
--max-children=150 --min-children=10
Our (1) mail server is configured like this:
CentOS 4.5
Exim 4.67
SpamAssassin version 3.2.3 running on Perl version 5.8.8
ClamAV 0.91.2 (saneSecurity updates)
- handles incoming/outgoing mail
- handles imap/pop/webmail request
Intel D Cpu 3.00Ghz with 2GB of Mem
80GB SATA root disk
200GB SATA mail disk (softraid mirror)
2xIntel e1000
Our mail server was taking a pounding on Friday,
Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBB
Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBB
Fri Sep 7 16:17:09 2007 [26914] info: prefork: child states: BBBBBBB
..snip...
..snip...
..snip...
Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBB
Fri Sep 7 16:17:19 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBIBBBBBBBBBBBBBBBBBBBBBBIBBB
..snip..
..snip..
Fri Sep 7 16:19:22 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBSBBSBB
Fri Sep 7 16:19:23 2007 [26914] info: prefork: child states:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBISBB
At the mist of the surge we had 95 child processess running, all busy!
Here are the sar memory stats...
kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree
kbswpused %swpused kbswpcad
16:10:02 16804 2056424 99.19 2900 1310880
2040036 208 0.01 0
16:20:10 37676 2035552 98.18 1872 237376 1736152
304092 14.90 78992
16:30:51 13924 2059304 99.33 1292 308944 1044160
996084 48.82 357444
16:40:02 76652 1996576 96.30 8208 1280796 1756236
284008 13.92 178696
Average: 26403 2046825 98.73 5880 1364057
2024199 16045 0.79 6152
Here are the warnings we saw in the spamd log...
Fri Sep 7 16:20:39 2007 [26914] info: prefork: child states:
IBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Fri Sep 7 16:20:40 2007 [25431] warn: spf: lookup failed: Can't locate
object method "new" via package "Net::DNS::RR::TXT" at
/xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/
DNS/RR.pm line 312.
Fri Sep 7 16:20:41 2007 [25428] warn: spf: lookup failed: Can't locate
object method "new" via package "Net::DNS::RR::TXT" at
/xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/
DNS/RR.pm line 312.
Fri Sep 7 16:22:18 2007 [24684] warn: plugin: eval failed: child
processing timeout at /xsys/sbin//spamd line 1246, <GEN683> line 3398.
Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in
pattern match (m//) at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 211,
<GEN749> line 3398.
Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in
scalar assignment at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 227,
<GEN749> line 3398.
Fri Sep 7 16:26:15 2007 [25227] info: spamd: clean message (1.5/5.0) for
cs242027:9190 in 406.1 seconds, 243776 bytes.
Fri Sep 7 16:26:19 2007 [24688] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:26:24 2007 [25046] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:26:28 2007 [24692] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:30:35 2007 [26914] info: spamd: server successfully spawned
child process, pid 26312
Fri Sep 7 16:30:37 2007 [24685] warn: spamd: copy_config timeout,
respawning child process after 1 messages at /xsys/sbin//spamd line 1103.
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: cannot ping 24702, file
handle not defined, child likely to still be processing SIGCHLD handler
after killing itself
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killing failed child 24702
fd=undefined at
/xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line
171.
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killed child 24702
Fri Sep 7 16:30:41 2007 [26914] info: spamd: handled cleanup of child pid
24702 due to SIGCHLD
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: cannot ping 24687, file
handle not defined, child likely to still be processing SIGCHLD handler
after killing itself
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killing failed child 24687
fd=undefined at
/xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line
171.
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killed child 24687
Looking at the swap usage, I was thinking I would be better if I reduced
the number of children processes and let thing queue up. I know I will
also have to look at exim and it's ratelimit command. Any other idea's on
handling spam surges/DoS?
Thanks
Paul
