Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
Sean Donelan wrote: What is the difference between a transit provider and an access provider, specially in the consumer space? Why is a transit provider expected to deliver the bits, but the access provider isn't? Since the bulk of Internet access is actually provided by wholesale providers (e.g. AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is the access provider and who is the transit provider? Both of these identities can and do exist within the same ISP. Their transit product would not include active mitigation, notification and filtering but the access provider part would. Pete
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
On Fri, 28 Nov 2003, Daniel Senie wrote: > At 06:24 PM 11/28/2003, Sean Donelan wrote: > > >On Sat, 29 Nov 2003, Petri Helenius wrote: > > > If you are an access provider, specially in the consumer space, you can > > > do many things to help the "Greater Internet" by keeping your own back > > > yard in good shape. In the transit business, you are expected to > > > deliver the bits regardless of the content so there the only viable > > > option is to drop packets where the source or destination addresses > > > don´t make sense. > > > >What is the difference between a transit provider and an access provider, > >specially in the consumer space? Why is a transit provider expected to > >deliver the bits, but the access provider isn't? Since the bulk of > >Internet access is actually provided by wholesale providers (e.g. > >AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is > >the access provider and who is the transit provider? > > And how do you handle the situation where a provider is both? UUNet, for > example, sells LOTS of T-1 lines to non-ISP businesses, and sells retail > dialup services to consumers. Sure they also sell wholesale bandwidth and > wholesale dialup services to ISPs, but it's not their whole business. > > The problem isn't "someone else's problem" for anyone. Clearly you can apply checks to parts of your network (stub bits) such as your dialup pool etc where can be sure about what src/dst addresses are fake, ie the access part of your business Steve
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
At 06:24 PM 11/28/2003, Sean Donelan wrote: On Sat, 29 Nov 2003, Petri Helenius wrote: > If you are an access provider, specially in the consumer space, you can > do many things to help the "Greater Internet" by keeping your own back > yard in good shape. In the transit business, you are expected to > deliver the bits regardless of the content so there the only viable > option is to drop packets where the source or destination addresses > don´t make sense. What is the difference between a transit provider and an access provider, specially in the consumer space? Why is a transit provider expected to deliver the bits, but the access provider isn't? Since the bulk of Internet access is actually provided by wholesale providers (e.g. AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is the access provider and who is the transit provider? And how do you handle the situation where a provider is both? UUNet, for example, sells LOTS of T-1 lines to non-ISP businesses, and sells retail dialup services to consumers. Sure they also sell wholesale bandwidth and wholesale dialup services to ISPs, but it's not their whole business. The problem isn't "someone else's problem" for anyone.
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
On Sat, 29 Nov 2003, Petri Helenius wrote: > If you are an access provider, specially in the consumer space, you can > do many things to help the "Greater Internet" by keeping your own back > yard in good shape. In the transit business, you are expected to > deliver the bits regardless of the content so there the only viable > option is to drop packets where the source or destination addresses > don´t make sense. What is the difference between a transit provider and an access provider, specially in the consumer space? Why is a transit provider expected to deliver the bits, but the access provider isn't? Since the bulk of Internet access is actually provided by wholesale providers (e.g. AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is the access provider and who is the transit provider?
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
Sean Donelan wrote: ISPs don't have (much) control over third-party computers. But they can control their network capacity. Of course, its not a complete solution. If you are a mid-level ISP, you may have a choke point to your customer but are vulnerable from your upstream provider. A better designed worm could impact even major backbones. If you are an access provider, specially in the consumer space, you can do many things to help the "Greater Internet" by keeping your own back yard in good shape. In the transit business, you are expected to deliver the bits regardless of the content so there the only viable option is to drop packets where the source or destination addresses don´t make sense. Pete
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
Hi, Sean. ] lower bandwidth<>higher bandwidth Great ASCII chart. :) ] Of course, there are some exceptions like a customer with an OC192 uplink ] or an ISP running a web hosting center on a ISDN link. Another bit to consider is address space. Code Red discovered a lot of folks with very small pipes (circa T1) and very large netblocks (circa /16). These folks paid a heavy price when hit with the "scan all IPs in the netblock" worms. Thanks, Rob. -- Rob Thomas http://www.cymru.com ASSERT(coffee != empty);
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
On Tue, 25 Nov 2003, Rob Thomas wrote: > Our choke points were always our peering or transit links. This > was the case for our (large) enterprise customers as well. Some people refer to it as the hourglass effect, but it has more than one bump. Generally only the smallest bottleneck controls the congestion. But worms and DDOS (but not DOS) violate some of the assumptions. lower bandwidth<>higher bandwidth Local Area Network (LAN) Campus Area Network Customer to ISP uplink ISP POP to Backbone ISP Intra-Backbone ISP to ISP transit/peer (same continent) Intercontinental circuits Of course, there are some exceptions like a customer with an OC192 uplink or an ISP running a web hosting center on a ISDN link.
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
Hi, Stuart. ] So you believe that the edges of the net are smaller, bandwidth-wise, ] than the core? This was certainly the case in my previous life at a large hosting provider. We had GigE LANs, used providers with OC192 backbones, but had only OC3 to OC12 links to our providers. Like most edge networks, we had CIRs on those uplinks that were considerably lower than the pipe size. A full OC12 turned out, at the time, to be darn expensive. :) Our choke points were always our peering or transit links. This was the case for our (large) enterprise customers as well. Thanks, Rob. -- Rob Thomas http://www.cymru.com ASSERT(coffee != empty);
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
On Monday, November 24, 2003, at 08:00 PM, Sean Donelan wrote: There are some natural choke points in the Internet between ISPs and customers. The customer may have a 1000 Mbps GigE LAN and the ISP may have an OC192 backbone, but the link between them is normally much smaller. Slammer, Blaster, etc had very little impact on the major ISP backbones, but did severaly congest some of the smaller choke points. Go ahead and ask UUNET, Sprint, AT&T, etc. what impact the worms had their networks. So you believe that the edges of the net are smaller, bandwidth-wise, than the core? So the (approximate) picture you would advocate would be that Slammer was rate limited at the customer/ISP interface? (I agree this is consistent with the fact that the tier-1s stayed up during Slammer). (I'm not trying to be difficult here - I'm just trying to figure out if we actually have any good understanding of this issue - and therefore any ability to predict what future worms might do to the Internet). (Blaster was not bandwidth limited so that's a whole different animal - it seems to have been limited by a slow scanning rate, and a poor transmission probability). Stuart. Stuart Staniford, President Tel: 707-840-9611 x 15 Silicon Defense - Worm Containment - http://www.silicondefense.com/ The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
On Mon, 24 Nov 2003, Stuart Staniford wrote: > So it would seem that worms are, at a minimum, not a simple or > unproblematic capacity management problem. Things are rarely as simple as they appear. Even buying a military grade black box may not solve the worm problem. There are some natural choke points in the Internet between ISPs and customers. The customer may have a 1000 Mbps GigE LAN and the ISP may have an OC192 backbone, but the link between them is normally much smaller. Slammer, Blaster, etc had very little impact on the major ISP backbones, but did severaly congest some of the smaller choke points. Go ahead and ask UUNET, Sprint, AT&T, etc. what impact the worms had their networks. ISPs don't have (much) control over third-party computers. But they can control their network capacity. Of course, its not a complete solution. If you are a mid-level ISP, you may have a choke point to your customer but are vulnerable from your upstream provider. A better designed worm could impact even major backbones.
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
Stuart Staniford writes: >I wasn't advocating a solution, just observing the way things would >have to be for worms to be purely a "buy a bigger box" problem (as I >think Sean was suggesting if I didn't misunderstand him). Ah. >It would generally seem that ISPs would provide more downstream >capacity than upstream, since this saves money and normally not all the >downstream customers will use all their bandwidth at the same time. Right; statistical multiplexing. >But a big worm could well break that last assumption. Yes, as could a number of events, but the response to a worm would probably be different from the latest streaming video event, or whatever. >So it would seem that worms are, at a minimum, not a simple or >unproblematic capacity management problem. Well, it would seem reasonable for an ISP to minimize a worm's effect on its non-worm customer traffic, and that might mean increasing capacity in some places, but I don't think the goal would be to move more worm traffic, but rather to reduce impact to other traffic. Presumably such activity would be combined with other anti-worm efforts.
Re: Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
On Monday, November 24, 2003, at 04:59 PM, [EMAIL PROTECTED] wrote: So, essentially, you are saying that the edges (customers, presumably) need to be bandwidth-limited to protect the core? I wasn't advocating a solution, just observing the way things would have to be for worms to be purely a "buy a bigger box" problem (as I think Sean was suggesting if I didn't misunderstand him). This tends to happen anyway due to statistical multiplexing, but is usually not what the customers would want if they considered the question, and is not what ISPs want if they bill by the bit. It would generally seem that ISPs would provide more downstream capacity than upstream, since this saves money and normally not all the downstream customers will use all their bandwidth at the same time. But a big worm could well break that last assumption. So it would seem that worms are, at a minimum, not a simple or unproblematic capacity management problem. Stuart. Stuart Staniford, President Tel: 707-840-9611 x 15 Silicon Defense - Worm Containment - http://www.silicondefense.com/ The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
Stuart Staniford writes: >It would seem for the Internet to reliably resist bandwidth attacks >from future worms, it has to be, roughly "bigger in the middle than at >the edges". If this is the case, then the worm can choke edges at the >sites it infects, but the rest of the net can still function. If it's >bigger at the edges than in the middle, you'd expect a big enough worm >would be able to choke the core. For a given ISP, you'd want capacity >to the upstream to be bigger than the capacity to downstream customers. > (It would seem like this would be the reverse of what economics would >tend to suggest). So, essentially, you are saying that the edges (customers, presumably) need to be bandwidth-limited to protect the core? This tends to happen anyway due to statistical multiplexing, but is usually not what the customers would want if they considered the question, and is not what ISPs want if they bill by the bit. >Do we really know much about the capacity of the Internet to carry worm >traffic? (We believe Slammer used a peak bandwidth of roughly 200 >Gbps). I suspect that in the end the main backbone constaint will be peering links, for larger ISPs.
Worm Bandwidth [was Re: Santa Fe city government computers knocked out by worm]
[Sorry for responding to old mail, but I'm catching up] On Sunday, November 16, 2003, at 02:12 PM, Sean Donelan wrote: I've often tried to explain that ISPs generally view worms as a "capacity planning" issue. Worms change the "eco-system" of the Internet and ISPs have to adapt. But ISPs generally can't "fix" the end-users or their computers. I'm curious to know if doing this is at all well understood? Those of us doing research on worm spread, I don't think have a completely clear understanding of the interaction of Internet bandwidth and worm spread. Slammer, we are pretty clear became bandwidth limited (the rate of spread slowed down dramatically about 40 seconds into the spread). But we don't really know where those chokepoints live (at the edge, or in the middle). It would seem for the Internet to reliably resist bandwidth attacks from future worms, it has to be, roughly "bigger in the middle than at the edges". If this is the case, then the worm can choke edges at the sites it infects, but the rest of the net can still function. If it's bigger at the edges than in the middle, you'd expect a big enough worm would be able to choke the core. For a given ISP, you'd want capacity to the upstream to be bigger than the capacity to downstream customers. (It would seem like this would be the reverse of what economics would tend to suggest). Do we really know much about the capacity of the Internet to carry worm traffic? (We believe Slammer used a peak bandwidth of roughly 200 Gbps). Stuart. Stuart Staniford, President Tel: 707-840-9611 x 15 Silicon Defense - Worm Containment - http://www.silicondefense.com/ The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Re: Santa Fe city government computers knocked out by worm
On Mon, Nov 17, 2003 at 09:40:01AM -0500, [EMAIL PROTECTED] said: > > Valdis Kletnieks responded: > > > > It doesn't take long for the average mechanic to learn that buying cheap > > > > wrenches is a bad idea. > > > > to which Alex replied: > > > Do you take your car to McLaren service center? Why not? They definitely > > > have better tools. > > > > To which I say: > > No, but if the mechanic I did go to had a habit of using tools that > > regularly caused my car to halt and catch fire with me in it, I think I'd > > switch mechanics until I found somebody that used more reliable tools. Alex said: > Again, for the end customer, the level of damage that they are experiencing > is too little to bother. I would definitely take exception to that statement, based only on the end users of Wintel machines I hear from (the rest of my family). They come fairly close to being average Windows end-users, have zero knowledge other than 'click here', and the latest round of worms and the truckload of critical security updates in the last month has had me on the phone trying to walk them through WindowsUpdate at least half a dozen times in the past week or two (as they're on dial-up, getting in from outside is a bit trickier). My dad threatens to turn the computer into a pile of twisted aluminum on a regular basis due to all the problems he's facing (some of which are due to the typical Windows user experience, some are due to worms, some to dialup, some to hardware (printer ink cartridges), all of which are exacerbated by lack of knowledge and experience). This is not what I would qualify as a level of damage "too little to bother" with. Shoulda bought them a Mac last year ... -- Scott Francis || darkuncle (at) darkuncle (dot) net illum oportet crescere me autem minui pgp0.pgp Description: PGP signature
Re: Santa Fe city government computers knocked out by worm
> Valdis Kletnieks responded: > > > It doesn't take long for the average mechanic to learn that buying cheap > > > wrenches is a bad idea. > > to which Alex replied: > > Do you take your car to McLaren service center? Why not? They definitely > > have better tools. > > To which I say: > No, but if the mechanic I did go to had a habit of using tools that regularly > caused my car to halt and catch fire with me in it, I think I'd switch > mechanics until I found somebody that used more reliable tools. Again, for the end customer, the level of damage that they are experiencing is too little to bother. Alex
Re: Santa Fe city government computers knocked out by worm
> > On Mon, 17 Nov 2003 06:26:50 EST, Alex Yuriev said: > > > > > Because for people outside our little industry the software is a tool to > > > get a JOB done, not the job itself. Valdis Kletnieks responded: > > It doesn't take long for the average mechanic to learn that buying cheap > > wrenches is a bad idea. to which Alex replied: > Do you take your car to McLaren service center? Why not? They definitely > have better tools. To which I say: No, but if the mechanic I did go to had a habit of using tools that regularly caused my car to halt and catch fire with me in it, I think I'd switch mechanics until I found somebody that used more reliable tools. -- Scott Francis || darkuncle (at) darkuncle (dot) net illum oportet crescere me autem minui pgp0.pgp Description: PGP signature
Re: Santa Fe city government computers knocked out by worm
On 17 Nov 2003, at 11:17, todd glassey wrote: H - I would have used a different picture - I would have said that "the average Ferrari Owner to realizes that if they don't tune their horse, it dies on them... while they are driving it.", So why don't the operators of Microsoft OS instances? - It's an avalanche of metaphors! I'm being buried alive!
Re: Santa Fe city government computers knocked out by worm
- Original Message - From: "Jeffrey Paul" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]>; "Alex Yuriev" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Monday, November 17, 2003 7:25 AM Subject: RE: Santa Fe city government computers knocked out by worm > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of [EMAIL PROTECTED] > Sent: Mon, 2003-11-17 10:23 > To: Alex Yuriev > Cc: [EMAIL PROTECTED] > Subject: Re: Santa Fe city government computers knocked out by worm > > > On Mon, 17 Nov 2003 06:26:50 EST, Alex Yuriev said: > > > Because for people outside our little industry the software > is a tool > > to get a JOB done, not the job itself. > > It doesn't take long for the average mechanic to learn that > buying cheap wrenches is a bad idea. Which is probably why they end up buying the expensive, supported one (like everyone else). It's also why they get worms. -j H - I would have used a different picture - I would have said that "the average Ferrari Owner to realizes that if they don't tune their horse, it dies on them... while they are driving it.", So why don't the operators of Microsoft OS instances? - Todd
Re: Santa Fe city government computers knocked out by worm
On Mon, 17 Nov 2003 15:25:08 GMT, Jeffrey Paul said: > Which is probably why they end up buying the expensive, supported one > (like everyone else). It's also why they get worms. I said cheap, not inexpensive. There's a difference. :) pgp0.pgp Description: PGP signature
Re: Santa Fe city government computers knocked out by worm
On Mon, 17 Nov 2003 06:26:50 -0500 (EST), you wrote: > >> >No explaination why Sante Fe officials had not patched the city's >> >computers in the three months since Microsoft announced the vulnerability >> >and released the software updates. Nor why Sante Fe didn't have up to >> >date anti-virus programs running on its computers. >> >> Nor why they were using such rubbish software for a mission- >> critical system. >> >Because for people outside our little industry the software is a tool to get >a JOB done, not the job itself. > >Alex A perceptive comment, but not actionable. This incident is what happens when non-tool oriented people must use tools. Our responsibility is to teach; theirs is to learn. Some people spend too much time sharpening their tools. Others too little. Neither is innocent when the job fails to get done. /kenw
Re: Santa Fe city government computers knocked out by worm
> On Mon, 17 Nov 2003 06:26:50 EST, Alex Yuriev said: > > > Because for people outside our little industry the software is a tool to get > > a JOB done, not the job itself. > > It doesn't take long for the average mechanic to learn that buying cheap > wrenches is a bad idea. Do you take your car to McLaren service center? Why not? They definitely have better tools. Alex
Re: Santa Fe city government computers knocked out by worm
In message <[EMAIL PROTECTED]>, Sean Donelan writes: > >The US is still losing relatively major city government computer networks >due to the Nachi/Welchia worm. > >Sante Fe city government's entire computer network was knocked offline >on Friday by the Nachi worm. City employees could not access e-mail or >work with their computers all day Friday, and the Santa Fe Public Library >was not able to access the Internet. > >Officials say the worm infected the system when an employee downloaded >music on a city computer. The article says the worm was able to infect >the city computer system by first disabling the system's virus detection >system. Both statements would be notable because known versions of >Nachi/Welchia don't spread that way. > >http://kobtv.com/index.cfm?viewer=storyviewer&id=6232&cat=HOME > >No explaination why Sante Fe officials had not patched the city's >computers in the three months since Microsoft announced the vulnerability >and released the software updates. Nor why Sante Fe didn't have up to >date anti-virus programs running on its computers. > I draw a different conclusion from the article: the channel from the techs who worked on it to the reporter was lossy... As you note, Nachi/ Welchia aren't spread by music downloads, nor do they disable AV software. I suspect that a Trojan'ed file-sharing program is more likely the culprit. --Steve Bellovin, http://www.research.att.com/~smb
RE: Santa Fe city government computers knocked out by worm
> -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of [EMAIL PROTECTED] > Sent: Mon, 2003-11-17 10:23 > To: Alex Yuriev > Cc: [EMAIL PROTECTED] > Subject: Re: Santa Fe city government computers knocked out by worm > > > On Mon, 17 Nov 2003 06:26:50 EST, Alex Yuriev said: > > > Because for people outside our little industry the software > is a tool > > to get a JOB done, not the job itself. > > It doesn't take long for the average mechanic to learn that > buying cheap wrenches is a bad idea. Which is probably why they end up buying the expensive, supported one (like everyone else). It's also why they get worms. -j
Re: Santa Fe city government computers knocked out by worm
On Mon, 17 Nov 2003 06:26:50 EST, Alex Yuriev said: > Because for people outside our little industry the software is a tool to get > a JOB done, not the job itself. It doesn't take long for the average mechanic to learn that buying cheap wrenches is a bad idea. pgp0.pgp Description: PGP signature
Re: Santa Fe city government computers knocked out by worm
> >No explaination why Sante Fe officials had not patched the city's > >computers in the three months since Microsoft announced the vulnerability > >and released the software updates. Nor why Sante Fe didn't have up to > >date anti-virus programs running on its computers. > > Nor why they were using such rubbish software for a mission- > critical system. > Because for people outside our little industry the software is a tool to get a JOB done, not the job itself. Alex
Re: Santa Fe city government computers knocked out by worm
On Sun, 16 Nov 2003, Jamie Reid wrote: > There was a comment (maybe even mine) in a previous thread > about accepting a base level of potentially compromised hosts > on a network, as the costs of rooting out every last one becomes > unwieldly. Networks are large enough that security must be > viewed as an economy of controls and risks instead of as a binary > state of secure or compromised. If your policy is not to root out every last one, then you need to beef up your network so a single compromised host doesn't bring down the whole network. The Internet is evidence that a network can continue to operate even with a very large number of compromised machines on a daily basis. On the other hand, if a single user downloading a music file on your network can take your entire network off the air for several days, you may have a problem. I've often tried to explain that ISPs generally view worms as a "capacity planning" issue. Worms change the "eco-system" of the Internet and ISPs have to adapt. But ISPs generally can't "fix" the end-users or their computers. System admins were able to completely eradicate the Morris worm. But most modern worms like Nimda, Code Red I/II, Slammer stick around. Sometimes a new worm like Nachi supplants an older worm like Blaster. Even if the ISP tries to be the great network firewall, we have mobile computers with mobile code. Laptops are too common, connecting to multiple networks.
Re: Santa Fe city government computers knocked out by worm
On Sun, 16 Nov 2003 06:22:08 -0500 (EST), Sean Donelan wrote: > >http://kobtv.com/index.cfm?viewer=storyviewer&id=6232&cat=HOME > >No explaination why Sante Fe officials had not patched the city's >computers in the three months since Microsoft announced the vulnerability >and released the software updates. Nor why Sante Fe didn't have up to >date anti-virus programs running on its computers. Nor why they were using such rubbish software for a mission- critical system.
Santa Fe city government computers knocked out by worm
The US is still losing relatively major city government computer networks due to the Nachi/Welchia worm. Sante Fe city government's entire computer network was knocked offline on Friday by the Nachi worm. City employees could not access e-mail or work with their computers all day Friday, and the Santa Fe Public Library was not able to access the Internet. Officials say the worm infected the system when an employee downloaded music on a city computer. The article says the worm was able to infect the city computer system by first disabling the system's virus detection system. Both statements would be notable because known versions of Nachi/Welchia don't spread that way. http://kobtv.com/index.cfm?viewer=storyviewer&id=6232&cat=HOME No explaination why Sante Fe officials had not patched the city's computers in the three months since Microsoft announced the vulnerability and released the software updates. Nor why Sante Fe didn't have up to date anti-virus programs running on its computers.
