Re: Dealing with infected users (Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
Vadim Antonov wrote: It should be pointed put that the ISPs have their share of blame for the quick-spreading worms, beause they neglected very simple precautions -- such as giving cutomers pre-configured routers or DSL/cable modems with firewalls disabled by default (instead of the standard end-user, let only outgoing connections thru configuration), and providing insufficient information to end-users on configuring these firewalls. And you´re willing to pay all the helpdesk persons helping these people to adjust their configurations to accommodate for KaZaa, BitTorrent, Quake3, Counter Strike, etc? It would be much easier and more centralized if the networking interfaces in operating systems would not expose services by default. But were already went there. Pete
ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
On Thu, 28 Aug 2003, Steve Carter wrote: The rate-limiters have become more interesting recently, meaning they've actually started dropping packets (quite a lot in some cases) because of the widespread exploitation of unpatched windows machines. Yep, the amount of ICMP traffic seems to be increasing on most backbones due to worm activity. It probably hasn't exceed HTTP yet, but it is surpasssing many other protocols. Some providers have seen ICMP increase by over 1,000% over the last two weeks. Unfortunately, the question sometimes becomes which packets do you care about more? Ping or HTTP? Patch your Windows boxes. Get your neighbors to patch their Windows boxes. Microsoft make a CD so people can fix their Windows machines before they connect them to the network.
Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
* Sean Donelan said: On Thu, 28 Aug 2003, Steve Carter wrote: The rate-limiters have become more interesting recently, meaning they've actually started dropping packets (quite a lot in some cases) because of the widespread exploitation of unpatched windows machines. Yep, the amount of ICMP traffic seems to be increasing on most backbones due to worm activity. It probably hasn't exceed HTTP yet, but it is surpasssing many other protocols. Some providers have seen ICMP increase by over 1,000% over the last two weeks. The results of our data collection is almost unbelievable. I've had to have it rechecked multiple times because I had a hard time even groking the scale. Like, dude, is your calculator broken? It appears that the volume is still growing ... even with the widespread publicity. Those of us that are sourcing this traffic need to protect ourselves and the community by rate limiting because the exploited are not. I agree with Wayne that we need to be smart (reads: very specific) about how we rate limit during this event. When the event is over we can go back to just a simple rate limit that protects us in a very general way until the next event jumps up. private message Yuh, Jay, I changed my tune ... you were right. /private message -Steve
Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
Inline. On Thu, Aug 28, 2003 at 12:01:16PM -0400, Sean Donelan said something to the effect of: On Thu, 28 Aug 2003, Steve Carter wrote: The rate-limiters have become more interesting recently, meaning they've actually started dropping packets (quite a lot in some cases) because of the widespread exploitation of unpatched windows machines. Yep, the amount of ICMP traffic seems to be increasing on most backbones due to worm activity. It probably hasn't exceed HTTP yet, but it is surpasssing many other protocols. Some providers have seen ICMP increase by over 1,000% over the last two weeks. I fear that all this has been a conspiracy machinated by an amalgam of coffee purveyors and aspirin/analgesic manufacturers. This is most definitely true. I work on GBLX's Internet Security team and had the dubious fortune of being the oncall engineer this week. The sheer volume of icmp I've see just as a result of slurping traffic off customer interfaces, not peering points, related to security incident reports is staggering. Facing facts, people are _not_ patching their stuff, in spite of pervasive pleas and warnings from vendors and media geeks. Many of the infected customers, presenting initially with symptoms of circuit saturation and latency, are shocked to learn that they are in effect DoSing themselves, and only then are they even mildly-motivated to seek out sub-par OS builds and patch their boxen. While a rate limit doesn't do anything to restore link health to those customers, it prevents them from flooding the playground for the rest of us. Others remain more or less clueless that they're throttling unholy quantities of icmp (among other things) until a node threatens to go unstable and we start filtering and swinging traffic in a flurry of damage control, subsequently calling _them_ and asking that the issue be investigated. Having a router reload or an upstream circuit become saturated is far more rigorous to the customers downstream than pruning back their capacity for icmp. We are operating in an unusual time, where these solutions may seem less than elegant, but are appropriate when overall network health and general responsibility dictate that more aggressive praxes of risk mitigation be deployed. When the din dies down to a more manageable roar, perhaps the caps can be re-evaluated. In the interim, these measures are levied in the name of customer/non-customer/device protection, and not enacted without great thought to the impact on our customers and downstreams. Unfortunately, the question sometimes becomes which packets do you care about more? Ping or HTTP? Unfortunate ultimatum, but cheers. It's true. Patch your Windows boxes. Get your neighbors to patch their Windows boxes. Simple, but brilliant. Please. If I could find my friggin fairy dust, I'd conjure up a trojan that went out and reloaded infected hosts with a new OS. Call it *poof*BSD perhaps? Just till this thing blows over... ;) Microsoft make a CD so people can fix their Windows machines before they connect them to the network. And this is a great idea... ymmv, --ra -- K. Rachael Treu rara at navigo dot com ..Fata viam invenient.. -- I am an employee of, but do not necessarily represent herein, Global Crossing, Ltd. --
Dealing with infected users (Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
We have been doing that. During quiet times our Customer Service Reps (CSR) are calling infected users telling them a) Their computer has been compromised. In its current state it can potentially be taken over by others or other users can look at the contents of their private files etc. b) It is currently interfering with other users connections. Particularly our DSL users can blast out at a fast enough rate to hamper dialup users. If the user is not home (often broadband users leave their computers on) the CSRs leave a message stating the customer can call in any time they like and they will be reactivated. Once doing so, they need to clean their machine ASAP-- there are several FREE point and click tools now. The majority comply and are understanding. I think the key is to emphasize that its in their best interest and that we did it for THEIR protection (i.e. someone can potentially take over your machine,look at your private files, delete things etc etc). Also emphasize that they need to be a responsible Internet participant -- e.g. how would they like it if another user was hampering their connection because that other user had a virus and we didnt get them to clean it up. Give your CSRs a script or talking points to follow and it should be smooth for the most part. ---Mike At 12:42 PM 28/08/2003 -0700, Dan Hollis wrote: On Thu, 28 Aug 2003, Rachael Treu wrote: Facing facts, people are _not_ patching their stuff, in spite of pervasive pleas and warnings from vendors and media geeks. There need to be more serious consequences for not patching. Like, having their ports turned down until they decide that patching might not be such a bad idea after all. -Dan -- [-] Omae no subete no kichi wa ore no mono da. [-]
Re: Dealing with infected users (Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
On Thu, 28 Aug 2003, Mike Tancsa wrote: The majority comply and are understanding. and the rest? -Dan -- [-] Omae no subete no kichi wa ore no mono da. [-]
Re: Dealing with infected users (Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
At 01:57 PM 28/08/2003 -0700, Dan Hollis wrote: On Thu, 28 Aug 2003, Mike Tancsa wrote: The majority comply and are understanding. and the rest? There will always be troublesome customers, but the VAST majority have been compliant. If they dont want to comply to something as reasonable as this, they will go to my competitors who will then have to deal with the flood of abuse hate mail (I am calling the FBI if you dont fix this), retaliatory attacks, black listings etc etc... i.e. they will become a headache for my competitors. Other sites who are large and dont necessarily have the resources to immediately find and kill the offending host (with sobig.f the headers will often show the NETBIOS name of the sending machine so its not THAT hard to find), we will add local rules to contain them for now until they have their IT consultants clean it up. But like I said before, give your CSRs a script. Explain to the customer how this is in their best interest... Most people are reasonable. We have all talked to people who say things like, I have had 10 different ISPs and none have made me do something like this! I demand... remember to ask yourself, why have they gone through 10 different ISPs . ---Mike
Re: Dealing with infected users (Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting
It should be pointed put that the ISPs have their share of blame for the quick-spreading worms, beause they neglected very simple precautions -- such as giving cutomers pre-configured routers or DSL/cable modems with firewalls disabled by default (instead of the standard end-user, let only outgoing connections thru configuration), and providing insufficient information to end-users on configuring these firewalls. --vadim
