Re: I'm getting killed with spammers
On Tuesday 17 October 2006 23:09, Bill Taroli wrote: Debbie D wrote: Last Mon, Tues Wed I had severe inflow of spam, always at 12.30p EST, Wed it didn't stop till almost 5p. The server seems to not be very cooperative when the queue grows over 200 or so. ... this high amount of spam, (BTW scoring at 20-well over 1000) is killing the loads and I have screaming clients.. I don't know that you're alone in seeing this increased traffic. For another domain I help manage, they were seeing a large influx of connections. For the most part, sender verification and RBL's were blocking them. But then they threw in a little twist... opening SMTP sessions and letting them sit. Open enough of these and processes build up (awaiting timeout) doing nothing and new connections fail -- a crude but effective DOS. Isn't this something Anvil is designed to handle? It seems SuSE installs this by default for postfix. I see log entries where is rate limits some IPs, usually when it looks like they are doing a dictionary job on me. The Postfix anvil(8) server maintains short-term statistics to defend against clients that hammer a server with either too many simultaneous sessions, or with too many successive requests within a configurable time interval. -- _ John Andersen
Re: I'm getting killed with spammers
John Andersen wrote: On Tuesday 17 October 2006 23:09, Bill Taroli wrote: Debbie D wrote: Last Mon, Tues Wed I had severe inflow of spam, always at 12.30p EST, Wed it didn't stop till almost 5p. The server seems to not be very cooperative when the queue grows over 200 or so. ... this high amount of spam, (BTW scoring at 20-well over 1000) is killing the loads and I have screaming clients.. I don't know that you're alone in seeing this increased traffic. For another domain I help manage, they were seeing a large influx of connections. For the most part, sender verification and RBL's were blocking them. But then they threw in a little twist... opening SMTP sessions and letting them sit. Open enough of these and processes build up (awaiting timeout) doing nothing and new connections fail -- a crude but effective DOS. Isn't this something Anvil is designed to handle? It seems SuSE installs this by default for postfix. I see log entries where is rate limits some IPs, usually when it looks like they are doing a dictionary job on me. The Postfix anvil(8) server maintains short-term statistics to defend against clients that hammer a server with either too many simultaneous sessions, or with too many successive requests within a configurable time interval. So that's what my firewall has been killing. I kept noticing timeout sessions with my mailserver (in the firewall log), and wondered why that was happening.
Re: I'm getting killed with spammers
On Wednesday 18 October 2006 00:50, nick wrote: So that's what my firewall has been killing. I kept noticing timeout sessions with my mailserver (in the firewall log), and wondered why that was happening. You should see anvil messages in mail log, but from the man page it is not at all clear that the firewall would be involved. It seems to handle this at the smtp server. -- _ John Andersen
Re: I'm getting killed with spammers
John Andersen wrote: On Wednesday 18 October 2006 00:50, nick wrote: So that's what my firewall has been killing. I kept noticing timeout sessions with my mailserver (in the firewall log), and wondered why that was happening. You should see anvil messages in mail log, but from the man page it is not at all clear that the firewall would be involved. It seems to handle this at the smtp server. Sorry, I was being a bit vague, I've got a stateful firewall between my mailserver and the external world, and I kept seeing that there were session timeouts no_connection_for_this_packet from a lot of different places. There's absolutely no problems with my connection or my mailserver load, and it was something that was leaving me a bit confused. I don't use anvil, at least not at the moment. A sort of moment of clarity. Nick
RE: I'm getting killed with spammers
Debbie D wrote: On Mon, October 16, 2006 2:28 pm, Debbie D said: this high amount of spam, (BTW scoring at 20-well over 1000) is killing the loads and I have screaming clients.. Just this afternoon (again around 12.30) it loaded up again with 312 mails.. the web based control panel was reacting so slow I would get 3 new ones for every one I managed to delete or deliver (I could not just delete the queue because some were actually valid mails in there) Server loads rose to well over 30, I shut exim - but cpanel was so kind to automagically restart it every time.. tried a reboot from ssh but that just hung.. the tech peeps did it from their end it it worked and brought the loads down so I could delete faster than they came in and now we're back to normal loads and queue I did upgrade to SA 3.1.7 last week - Wed night after a long day of battling the loads.. and that seemed to go well suggestions? Offers of help??? You probably have max children set too high. When a big bunch of messages come in, they all run, you don't have enough memory, and your system starts swapping like crazy. That brings everything on your server to a near halt. It reduces throughput, which means you get a backlog, which means you get stuck in this state because all the children stay active hogging RAM and trying to process the backlog. The solution is to either expand the RAM so the system can really handle that many active children at once, or set the maximum number of children to something much lower. Try 2 or 3 even. It seems like more children would mean more work getting done, and that's true, but it's only true up to a point, and you've passed that point. OK Logan I will investigate the RAM and see if it needs to be up'd and kick the maxchild back down to 10 in the mean time.. 10 is still a fairly large number of SA processes. I don't know if you mentioned how much RAM this machine has, but if you allow 50MB for each child, that's 500MB just for SA (this will vary depending on your settings and add-on rules). If you don't have that much available physical RAM, the system will start swapping and things will get very slow. On my system with 1GB RAM, I have to keep max-children down to 4 to avoid problems. This system also runs the mailserver, ClamAV, and Bind. I could probably get away with more processes if it was just doing SA. -- Bowie
RE: I'm getting killed with spammers
Title: RE: I'm getting killed with spammers FWIW: I think traffic is up all around. I'm seeing a definite increase in the past weeks. Most are being stopped via RBLs, but there is a lot more to stop! --Chris
Re: I'm getting killed with spammers
Chris Santerre wrote: FWIW: I think traffic is up all around. I'm seeing a definite increase in the past weeks. Most are being stopped via RBLs, but there is a lot more to stop! I'll second that, my account alone has been getting around 230 caught spams a day and quite a good few missed, it seems to be down to a rather rapid change in the content of the spam, just as the spam filters adapt and learn to the point of blocking spam X it stops and spam y kicks in! In the last few weeks i've added spamhaus and surbs tests, raised scores and lowered thresholds and i'm still getting notably higher volumes of spam (on all my accounts in fact!!) Tis an epidemic I tells ya! -- Mike Woods Systems Administrator
Re: I'm getting killed with spammers
On Mon, October 16, 2006 2:28 pm, Debbie D said: this high amount of spam, (BTW scoring at 20-well over 1000) is killing the loads and I have screaming clients.. Just this afternoon (again around 12.30) it loaded up again with 312 mails.. the web based control panel was reacting so slow I would get 3 new ones for every one I managed to delete or deliver (I could not just delete the queue because some were actually valid mails in there) Server loads rose to well over 30, I shut exim - but cpanel was so kind to automagically restart it every time.. tried a reboot from ssh but that just hung.. the tech peeps did it from their end it it worked and brought the loads down so I could delete faster than they came in and now we're back to normal loads and queue I did upgrade to SA 3.1.7 last week - Wed night after a long day of battling the loads.. and that seemed to go well suggestions? Offers of help??? At this point, you probably need to find some way to blacklist part of that load, to keep your server from dealing with it. It may be possible to improve SA performance so that you can survive the onslaught, but SA does mean that your server has to do something with each email it scans. A 'quick fix' would actually be to turn SA off. The (spam) messages will all go through, but it should mean less load on your system. Look through the spam sent in those bursts and see if there is any way you can identify them *quickly*, preferably by IP addresses. Then block them so your server doesn't have to deal with them. Daniel T. Staal Daniel I have tried that but apparently they are coming from everywhere all at once.. I did find one that was really bad and blocked it with IPtables.. but that one continues to show up in my log watch where I would think it would go away with the entry.. client 12.130.132.229 error sending response: host unreachable: 853 Time(s) and that is a LOW number for this guy.. it some days its up to 2000 I traced this and it is a an ATT IP for some kind of business service they offer You probably have max children set too high. When a big bunch of messages come in, they all run, you don't have enough memory, and your system starts swapping like crazy. That brings everything on your server to a near halt. It reduces throughput, which means you get a backlog, which means you get stuck in this state because all the children stay active hogging RAM and trying to process the backlog. The solution is to either expand the RAM so the system can really handle that many active children at once, or set the maximum number of children to something much lower. Try 2 or 3 even. It seems like more children would mean more work getting done, and that's true, but it's only true up to a point, and you've passed that point. - Logan OK Logan I will investigate the RAM and see if it needs to be up'd and kick the maxchild back down to 10 in the mean time.. the other thing I did last week was Number of minutes between mail server queue runs (default is 60).: I lowered it to 90 minutes from 4 hours but obviously that didn't help one bit Is the mail legitimate email? Meaning does the email come from wherever to *valid email addresses* on the server or do you have a system that will catch everything at the smtp level and then sort it out later? If your server catches everything, the smtp gate should probably be fortified with greylisting and invalid email address rejection first. There is not enough other info for me to recommend further... Thanks and kind regards, - rh 99% of the 300+ mails today and last week were addressed to valid users but I'd say 60%+ was truly spam.. today as I manually delivered from Cpanel's WHM individually, I tailed the maillog and many of them were scored and trashed.. but with that said there was several very valid mails to very valid users.. I have the whole machine set to fail for invalid users which everyone on the cpanel forums say is much more efficient than blackhole
Re: I'm getting killed with spammers
On Mon, October 16, 2006 2:28 pm, Debbie D said: this high amount of spam, (BTW scoring at 20-well over 1000) is killing the loads and I have screaming clients.. Just this afternoon (again around 12.30) it loaded up again with 312 mails.. the web based control panel was reacting so slow I would get 3 new ones for every one I managed to delete or deliver (I could not just delete the queue because some were actually valid mails in there) Server loads rose to well over 30, I shut exim - but cpanel was so kind to automagically restart it every time.. tried a reboot from ssh but that just hung.. the tech peeps did it from their end it it worked and brought the loads down so I could delete faster than they came in and now we're back to normal loads and queue I did upgrade to SA 3.1.7 last week - Wed night after a long day of battling the loads.. and that seemed to go well suggestions? Offers of help??? At this point, you probably need to find some way to blacklist part of that load, to keep your server from dealing with it. It may be possible to improve SA performance so that you can survive the onslaught, but SA does mean that your server has to do something with each email it scans. A 'quick fix' would actually be to turn SA off. The (spam) messages will all go through, but it should mean less load on your system. Look through the spam sent in those bursts and see if there is any way you can identify them *quickly*, preferably by IP addresses. Then block them so your server doesn't have to deal with them. Daniel T. Staal --- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. ---
Re: I'm getting killed with spammers
On Mon, 16 Oct 2006, Debbie D wrote: I have max child set to 15 (up from 5) and not sure what else I can offer in the way of what you need to know to help me, but if you tell me where to look I can spout what you need. : : Just this afternoon (again around 12.30) it loaded up again with 312 mails.. the web based control panel was reacting so slow I would get 3 new ones for every one I managed to delete or deliver (I could not just delete the queue because some were actually valid mails in there) Server loads rose to well over 30, I shut exim You probably have max children set too high. When a big bunch of messages come in, they all run, you don't have enough memory, and your system starts swapping like crazy. That brings everything on your server to a near halt. It reduces throughput, which means you get a backlog, which means you get stuck in this state because all the children stay active hogging RAM and trying to process the backlog. The solution is to either expand the RAM so the system can really handle that many active children at once, or set the maximum number of children to something much lower. Try 2 or 3 even. It seems like more children would mean more work getting done, and that's true, but it's only true up to a point, and you've passed that point. - Logan
RE: I'm getting killed with spammers
I need some help here.. Last Mon, Tues Wed I had severe inflow of spam, always at 12.30p EST, Wed it didn't stop till almost 5p. The server seems to not be very cooperative when the queue grows over 200 or so. I have max child set to 15 (up from 5) and not sure what else I can offer in the way of what you need to know to help me, but if you tell me where to look I can spout what you need. The install is out of the box with few if any mods except exim does have the dictionary attack, I run BFD and APF I do not believe I have been hacked into.. I DO read the logwatch daily and do poke around looking for dropped files on a semi regular basis.. this high amount of spam, (BTW scoring at 20-well over 1000) is killing the loads and I have screaming clients.. Just this afternoon (again around 12.30) it loaded up again with 312 mails.. the web based control panel was reacting so slow I would get 3 new ones for every one I managed to delete or deliver (I could not just delete the queue because some were actually valid mails in there) Server loads rose to well over 30, I shut exim - but cpanel was so kind to automagically restart it every time.. tried a reboot from ssh but that just hung.. the tech peeps did it from their end it it worked and brought the loads down so I could delete faster than they came in and now we're back to normal loads and queue I did upgrade to SA 3.1.7 last week - Wed night after a long day of battling the loads.. and that seemed to go well suggestions? Offers of help??? thanks Debbie, Is the mail legitimate email? Meaning does the email come from wherever to *valid email addresses* on the server or do you have a system that will catch everything at the smtp level and then sort it out later? If your server catches everything, the smtp gate should probably be fortified with greylisting and invalid email address rejection first. There is not enough other info for me to recommend further... Thanks and kind regards, - rh -- Robert - Abba Communications Computer Internet Services (509) 624-7159 - www.abbacomm.net
