Re: Cascading Failures Could Crash the Global Internet
From: Stewart, William C (Bill), SALES I think the key is that the failures described in the paper are caused by overload rather than other things - too much demand for power blows out the generator, and without it, the grid tries to get the power from the next nearest generators, which overload and fail, and try to pull an even large amount from the _next_ nearest, etc. So the bit about heterogeneity is probably referring to the fact that some nodes are bigger or better-connected than others, and are more likely to blow out a bunch of their neighbors when they fail and shed a big load. That's not really how Internet systems usually fail. A prime example of this theory was the large network I was using back when IE5 first came out. They had one circuit bad which overloaded an ATM circuit at another NAP causing it to generate bit errors. Shutting down the second circuit overloaded both MAE circuits effectively shutting down the network. However, it required manual intervention to create full failure, otherwise TCP would pull back to being useless, effectively killing all connections going that path, but not causing an issue with other paths until the manual intervention of shutting down the cirucit. While in theory it was still a cascade failure, it was also poor planning/policy on the part of the network to not be able to compensate in case of failure. The information provided may be partially inaccurate and is only hearsay concerning actual outages and effects when various interventions were tried; no hard fact. Thus it could be taken as solely my conjecture and not actual fact. -Jack
Re: Cascading Failures Could Crash the Global Internet
Hello; A packet switched network can be engineered against cascading failures in a way that's hard for a circuit switched network. Every time you see a random wait in a protocol, it's a good bet that the protocol writers were trying to protect against the tight coupling that leads to cascading failures. Regards Marshall Eubanks On Sunday, February 9, 2003, at 10:07 AM, Jack Bates wrote: From: Stewart, William C (Bill), SALES I think the key is that the failures described in the paper are caused by overload rather than other things - too much demand for power blows out the generator, and without it, the grid tries to get the power from the next nearest generators, which overload and fail, and try to pull an even large amount from the _next_ nearest, etc. So the bit about heterogeneity is probably referring to the fact that some nodes are bigger or better-connected than others, and are more likely to blow out a bunch of their neighbors when they fail and shed a big load. That's not really how Internet systems usually fail. A prime example of this theory was the large network I was using back when IE5 first came out. They had one circuit bad which overloaded an ATM circuit at another NAP causing it to generate bit errors. Shutting down the second circuit overloaded both MAE circuits effectively shutting down the network. However, it required manual intervention to create full failure, otherwise TCP would pull back to being useless, effectively killing all connections going that path, but not causing an issue with other paths until the manual intervention of shutting down the cirucit. While in theory it was still a cascade failure, it was also poor planning/policy on the part of the network to not be able to compensate in case of failure. The information provided may be partially inaccurate and is only hearsay concerning actual outages and effects when various interventions were tried; no hard fact. Thus it could be taken as solely my conjecture and not actual fact. -Jack
Re: Cascading Failures Could Crash the Global Internet
Homogenous, in this context, does not mean similar platform connectivity, but nodes with same degree connecting to each other. Ah, that makes it more clear. So a full mesh would be better? ;-) no, fine grained peering would be better. -- Paul Vixie
Re: Cascading Failures Could Crash the Global Internet
The model proposed makes several assumptions. My question is about: Many real-world networks are heterogeneous and as such are expected to undergo large-scale cascades if some vital nodes are attacked. on page 3. I do not get the basis for this assumption. So any help for a 60's educated math major would be appreciated. On Thu, 6 Feb 2003 [EMAIL PROTECTED] wrote: The paper is avaibable on the Los Alamos site free of charge: http://xxx.lanl.gov/PS_cache/cond-mat/pdf/0301/0301086.pdf - Original Message - From: Sean Donelan [EMAIL PROTECTED] Date: Thursday, February 6, 2003 12:43 pm Subject: Cascading Failures Could Crash the Global Internet Sigh, there are differences between tightly coupled networks, such as the electric power grid and loosely couple networks like the Internet. But there are also some similarities, such as electric grids use DC interconnections to limit how far AC disturbances propagate; the Internet uses AS interconnections to limit IGP disturbances from propagating. http://sci.newsfactor.com/perl/story/20686.html The actual article requires payment to read http://ojps.aip.org/getabs/servlet/GetabsServlet? prog=normalid=PLEEE8660606510201idtype=cvipsgifs=Yes _ Douglas Denault [EMAIL PROTECTED] Voice: 301-469-8766 Fax: 301-469-0601
Re: Cascading Failures Could Crash the Global Internet
I believe the comments about heterogenous networks has to do with a measurement called assortivitiy that is used in statistical mechanics. A homogenous network is when nodes connect preferentially to nodes like them. In a heterogenous network they connect to nodes that are not like them. For networks like the Intneret and the electric grid it is measured by the number of connections a node has. The kicker, that the author's are alluding to, is that the more heterogenous a network is the more vulnerable it is to targeted attack. By taking out a highly connected node - lots of poorly connected nodes that use it as a hub are lost. The AS network had the highest heterogenous score of real-world tested networks, so lots of folks on that bandwagon. That said I don't think the tolerance parameter they set up in the paper makes much sense when applied to the Internet at the AS level. Basically it says once traffic exceeds a certain threshold the node will fail and cause cascades across the network. You guys are the experts but that does not sound overly realistic. - Original Message - From: Douglas Denault [EMAIL PROTECTED] Date: Saturday, February 8, 2003 7:31 am Subject: Re: Cascading Failures Could Crash the Global Internet The model proposed makes several assumptions. My question is about: Many real-world networks are heterogeneous and as such are expected to undergo large-scale cascades if some vital nodes are attacked. on page 3. I do not get the basis for this assumption. So any help for a 60's educated math major would be appreciated. On Thu, 6 Feb 2003 [EMAIL PROTECTED] wrote: The paper is avaibable on the Los Alamos site free of charge: http://xxx.lanl.gov/PS_cache/cond-mat/pdf/0301/0301086.pdf - Original Message - From: Sean Donelan [EMAIL PROTECTED] Date: Thursday, February 6, 2003 12:43 pm Subject: Cascading Failures Could Crash the Global Internet Sigh, there are differences between tightly coupled networks, such as the electric power grid and loosely couple networks like the Internet. But there are also some similarities, such as electric grids use DC interconnections to limit how far AC disturbances propagate; the Internet uses AS interconnections to limit IGP disturbances from propagating. http://sci.newsfactor.com/perl/story/20686.html The actual article requires payment to read http://ojps.aip.org/getabs/servlet/GetabsServlet? prog=normalid=PLEEE8660606510201idtype=cvipsgifs=Yes _ Douglas Denault [EMAIL PROTECTED] Voice: 301-469-8766 Fax: 301-469-0601
Re: Cascading Failures Could Crash the Global Internet
sgorman1 == sgorman1 [EMAIL PROTECTED] writes: sgorman1 I believe the comments about heterogenous networks has to do sgorman1 with a measurement called assortivitiy that is used in sgorman1 statistical mechanics. A homogenous network is when nodes sgorman1 connect preferentially to nodes like them. In a sgorman1 heterogenous network they connect to nodes that are not like sgorman1 them. For networks like the Intneret and the electric grid sgorman1 it is measured by the number of connections a node has. sgorman1 The kicker, that the author's are alluding to, is that the sgorman1 more heterogenous a network is the more vulnerable it is to sgorman1 targeted attack. By taking out a highly connected node - sgorman1 lots of poorly connected nodes that use it as a hub are sgorman1 lost. The AS network had the highest heterogenous score of sgorman1 real-world tested networks, so lots of folks on that sgorman1 bandwagon. I don't see how the fact that a network is homogeneous or heterogeneous has anything to do with how well connected it is. The only possible sense to this I can see is that, statistically, you are more likely to have a platform that the attacker has a viable attack for if you have lots of different platforms. But at the same time, if the attacker only has one exploit (or whatever attack vector), then you are also in a MUCH better position than someone who's network is made up 100% of that platform. I'm still not sure how having a homogeneous network helps. Either you aren't explaining it well, or I'm being stupid. I consider both possibilities to be equally likely at this point. ;-) IMHO, Michael
Re: Cascading Failures Could Crash the Global Internet
I believe the answer meant heterogenous has a meaning in a statistical context. As I was a Real Variables guy I, was weak on statistics (of my day). Math guys love to use perfectly good English words giving them different meanings. Assuming that the given definition is correct, the applicability of the assumption to the backbone is still not clear to me. While not doubting the mathematical model, it seems to me there is little empirical evidence to support it in this context. Or I am in the second half of your second point. On 8 Feb 2003, Michael Lamoureux wrote: sgorman1 == sgorman1 [EMAIL PROTECTED] writes: sgorman1 I believe the comments about heterogenous networks has to do sgorman1 with a measurement called assortivitiy that is used in sgorman1 statistical mechanics. A homogenous network is when nodes sgorman1 connect preferentially to nodes like them. In a sgorman1 heterogenous network they connect to nodes that are not like sgorman1 them. For networks like the Intneret and the electric grid sgorman1 it is measured by the number of connections a node has. sgorman1 The kicker, that the author's are alluding to, is that the sgorman1 more heterogenous a network is the more vulnerable it is to sgorman1 targeted attack. By taking out a highly connected node - sgorman1 lots of poorly connected nodes that use it as a hub are sgorman1 lost. The AS network had the highest heterogenous score of sgorman1 real-world tested networks, so lots of folks on that sgorman1 bandwagon. I don't see how the fact that a network is homogeneous or heterogeneous has anything to do with how well connected it is. The only possible sense to this I can see is that, statistically, you are more likely to have a platform that the attacker has a viable attack for if you have lots of different platforms. But at the same time, if the attacker only has one exploit (or whatever attack vector), then you are also in a MUCH better position than someone who's network is made up 100% of that platform. I'm still not sure how having a homogeneous network helps. Either you aren't explaining it well, or I'm being stupid. I consider both possibilities to be equally likely at this point. ;-) IMHO, Michael _ Douglas Denault [EMAIL PROTECTED] Voice: 301-469-8766 Fax: 301-469-0601
Re: Cascading Failures Could Crash the Global Internet
Ya - that pretty much sums it up - interesting model dubious application in reality. Homogenous, in this context, does not mean similar platform connectivity, but nodes with same degree connecting to each other. It is more likely for a node with ten connections to connect to other nodes with ten connections than one with one or two connections. The Internet is heterogenous. So, take UUnet as your nodes at the AS level and lets say it has 1700 or so connections to other AS's. Most of those other AS's would not not also have 1700 connections, thus it is heterogenous. Lots of problems in applying that to cascading failures in reality - making AS701 disappear, BGP allowing a cascading traffic failure from having to redistribute AS701's traffic etc etc. Hope that makes it more clear, but it is just my interpretation of someone elses work. - Original Message - From: Douglas Denault [EMAIL PROTECTED] Date: Saturday, February 8, 2003 3:23 pm Subject: Re: Cascading Failures Could Crash the Global Internet I believe the answer meant heterogenous has a meaning in a statistical context. As I was a Real Variables guy I, was weak on statistics (of my day). Math guys love to use perfectly good English words giving them different meanings. Assuming that the given definition is correct, the applicability of the assumption to the backbone is still not clear to me. While not doubting the mathematical model, it seems to me there is little empirical evidence to support it in this context. Or I am in the second half of your second point. On 8 Feb 2003, Michael Lamoureux wrote: sgorman1 == sgorman1 [EMAIL PROTECTED] writes: sgorman1 I believe the comments about heterogenous networks has to do sgorman1 with a measurement called assortivitiy that is used in sgorman1 statistical mechanics. A homogenous network is when nodes sgorman1 connect preferentially to nodes like them. In a sgorman1 heterogenous network they connect to nodes that are not like sgorman1 them. For networks like the Intneret and the electric grid sgorman1 it is measured by the number of connections a node has. sgorman1 The kicker, that the author's are alluding to, is that the sgorman1 more heterogenous a network is the more vulnerable it is to sgorman1 targeted attack. By taking out a highly connected node - sgorman1 lots of poorly connected nodes that use it as a hub are sgorman1 lost. The AS network had the highest heterogenous score of sgorman1 real-world tested networks, so lots of folks on that sgorman1 bandwagon. I don't see how the fact that a network is homogeneous or heterogeneous has anything to do with how well connected it is. The only possible sense to this I can see is that, statistically, you are more likely to have a platform that the attacker has a viable attack for if you have lots of different platforms. But at the same time, if the attacker only has one exploit (or whatever attack vector), then you are also in a MUCH better position than someone who's network is made up 100% of that platform. I'm still not sure how having a homogeneous network helps. Either you aren't explaining it well, or I'm being stupid. I consider both possibilities to be equally likely at this point. ;-) IMHO, Michael _ Douglas Denault [EMAIL PROTECTED] Voice: 301-469-8766 Fax: 301-469-0601
Re: Cascading Failures Could Crash the Global Internet
I think the key is that the failures described in the paper are caused by overload rather than other things - too much demand for power blows out the generator, and without it, the grid tries to get the power from the next nearest generators, which overload and fail, and try to pull an even large amount from the _next_ nearest, etc. So the bit about heterogeneity is probably referring to the fact that some nodes are bigger or better-connected than others, and are more likely to blow out a bunch of their neighbors when they fail and shed a big load. That's not really how Internet systems usually fail. Overload can cause problems, and we've seen congestion collapse in the past, but TCP is usually tuned to discourage it; when a system is overloaded, well-behaved applications (which is most of them) back off, gradually or rapidly, but unless the load is weird enough to blow out router CPUs or crowd out BGP and OSPF packets, usually the network itself stays up and running. If what's failing is an overload of BGP routes or something, that's different - and sometimes the load on the system shrinks as components fail, but sometimes that just makes everything flap all at once, increasing load and delaying convergence.
Re: Cascading Failures Could Crash the Global Internet
sgorman1 == sgorman1 [EMAIL PROTECTED] writes: sgorman1 Homogenous, in this context, does not mean similar platform sgorman1 connectivity, but nodes with same degree connecting to each sgorman1 other. Ah, that makes it more clear. So a full mesh would be better? ;-) sgorman1 Lots of problems in applying that to cascading failures in sgorman1 reality - making AS701 disappear, BGP allowing a cascading sgorman1 traffic failure from having to redistribute AS701's sgorman1 traffic etc etc. Hope that makes it more clear, but it is sgorman1 just my interpretation of someone elses work. Yeah, I have all sorts of problems with applying this model to reality. But then, we already determined that I was being stupid. ;-) thanks, Michael
Re: Cascading Failures Could Crash the Global Internet
--- Stewart, William C (Bill), SALES
[EMAIL PROTECTED] wrote:
If what's failing is an overload of BGP routes or
something,
that's different - and sometimes the load on the
system shrinks
as components fail, but sometimes that just makes
everything
flap all at once, increasing load and delaying
convergence.
I seem to recall a massive routing failure in October
which was caused by BGP getting imported to a major
ISPs IGP...
The core ${VENDOR 1}routers were able to handle the
influx of routes, but the edge ${VENDOR 2} routers
could not handle the influx - so the failure didn't
exactly cascade, but did more of a ripple. However,
the reloading of all of the edge devices increased the
BGP instability.
-David Barak
-fully RFC 1925 compliant-
__
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
Re: Cascading Failures Could Crash the Global Internet
N. Richard Solis wrote: Yeah yeah yeah. I know that everything isn't simple. I actually worked at a power plant so none of this is new to me. Can cascading failures occur? Yes. Witness the Great Blackout in NYC. My point was that there are places where the electrical network is designed to "blow the bolts" to TRY and protect everything. Does it work? Most of the time, yes. All of the time? NO. Bringing this back to topic. What you are refering to is similar to a failure within an AS. When you start having problems within one section of your network that could jeapordize the rest of your network, you cut it off until the problem can be fixed. Does it work? Most of the time, yes. All the time? No. Sometimes the failure is too rapid to avoid the cascade failure within the AS. This practice is seprate from grid and AS interconnects. It is a complicated problem but you'd be suprised at how fast things can happen when you HAVE to keep the system running. There is a tremendous amount of skill concentrated in that field and they do a good job of keeping everything running well. How many turbine overspeed events do snip I agree. The same can be said for many networks. The difference is that dealing with some networking problems is new to many engineers. Without proper training and expecting a cascade failure, how do you know the fastest method to deal with it? I've had lots of practice on my network. I have an average stabalization rate of about 5 minutes now, but then, I redesigned my network a long time ago to effectively deal with such problems in a shorter time span. The loss of a single transmission line isn't going to cause a whole station to trip. If you're losing a bunch though, you've probably got lots of other problems to worry about. Also true with many networks today. However, this topic falls within a single grid. The original analogy was dealing with grid interconnects which have different requirements and must be protected at all costs. IF, and I don't think it's happened in a very long time, an entire grid lost integrity, it would be unacceptable for the grid to cascade into the other two grids. Extra percautions are put into place. In the same reguards, many Autonomous Systems do have different policies reguarding their interconnects compared to their internal network. -Jack
Re: Cascading Failures Could Crash the Global Internet
The paper is avaibable on the Los Alamos site free of charge: http://xxx.lanl.gov/PS_cache/cond-mat/pdf/0301/0301086.pdf - Original Message - From: Sean Donelan [EMAIL PROTECTED] Date: Thursday, February 6, 2003 12:43 pm Subject: Cascading Failures Could Crash the Global Internet Sigh, there are differences between tightly coupled networks, such as the electric power grid and loosely couple networks like the Internet. But there are also some similarities, such as electric grids use DC interconnections to limit how far AC disturbances propagate; the Internet uses AS interconnections to limit IGP disturbances from propagating. http://sci.newsfactor.com/perl/story/20686.html The actual article requires payment to read http://ojps.aip.org/getabs/servlet/GetabsServlet? prog=normalid=PLEEE8660606510201idtype=cvipsgifs=Yes
Re: Cascading Failures Could Crash the Global Internet
I don't know of too many electrical distribution networks that use DC interconnection to limit AC failures from propogating. The main cause of AC disruption is a power plant getting out of phase with the rest of the power plants on the grid. When that happens, the plant "trips" of goes off-line to protect the entire grid. You lose some generating capacity but you dont fry everything on the network either. http://www.nerc.com/ There are some states that operate their own grids. Texas, for example. -Richard Sean Donelan wrote: Sigh, there are differences between tightly coupled networks, such as the electric power grid and loosely couple networks like the Internet. But there are also some similarities, such as electric grids use DC interconnections to limit how far AC disturbances propagate; the Internet uses AS interconnections to limit IGP disturbances from propagating. http://sci.newsfactor.com/perl/story/20686.html The actual article requires payment to read http://ojps.aip.org/getabs/servlet/GetabsServlet?prog=normalid=PLEEE8660606510201idtype=cvipsgifs=Yes
Re: Cascading Failures Could Crash the Global Internet
On Thu, 6 Feb 2003, N. Richard Solis wrote: I don't know of too many electrical distribution networks that use DC interconnection to limit AC failures from propogating. There are 3 power grids in the continental US: Western, Eastern, and Texas. (http://www.eia.doe.gov/cneaf/electricity/page/fact_sheets/transmission.html) These are interconnected using high voltage DC ties, for reasons including (but not limited to) what Sean said. - d. -- Dominic J. Eidson Baruk Khazad! Khazad ai-menu! - Gimli --- http://www.the-infinite.org/ http://www.the-infinite.org/~dominic/
Re: Cascading Failures Could Crash the Global Internet
RFC 3439 talks of coupling amplification and it's relation to the Internet. -danny Sigh, there are differences between tightly coupled networks, such as the electric power grid and loosely couple networks like the Internet. But there are also some similarities, such as electric grids use DC interconnections to limit how far AC disturbances propagate; the Internet uses AS interconnections to limit IGP disturbances from propagating. http://sci.newsfactor.com/perl/story/20686.html The actual article requires payment to read http://ojps.aip.org/getabs/servlet/GetabsServlet?prog=normalid=PLEEE8660606510201idtype=cvipsgifs=Yes
Re: Cascading Failures Could Crash the Global Internet
BGP flap limiting may be correlated to this action... the good thing about packets is that they don't require energy to be dropped; electricity needs to be consumed somewhere, probably generating heat. Rubens - Original Message - From: N. Richard Solis [EMAIL PROTECTED] To: Sean Donelan [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Thursday, February 06, 2003 7:09 PM Subject: Re: Cascading Failures Could Crash the Global Internet | I don't know of too many electrical distribution networks that use DC interconnection to limit AC failures from propogating. | | The main cause of AC disruption is a power plant getting out of phase with the rest of the power plants on the grid. When that happens, the plant trips of goes off-line to protect the entire grid. You lose some generating capacity but you dont fry everything on the network either. | | http://www.nerc.com/ | | There are some states that operate their own grids. Texas, for example. | | -Richard | | | Sean Donelan wrote: | | | | Sigh, there are differences between tightly coupled networks, such as | the electric power grid and loosely couple networks like the Internet. | But there are also some similarities, such as electric grids use DC | interconnections to limit how far AC disturbances propagate; the | Internet uses AS interconnections to limit IGP disturbances from | propagating. | | http://sci.newsfactor.com/perl/story/20686.html | | The actual article requires payment to read | http://ojps.aip.org/getabs/servlet/GetabsServlet?prog=normalid=PLEEE866 0606510201idtype=cvipsgifs=Yes | | |
Re: Cascading Failures Could Crash the Global Internet
On Thu, 6 Feb 2003, N. Richard Solis wrote: The main cause of AC disruption is a power plant getting out of phase with the rest of the power plants on the grid. This is typically a result of sudden load change (loss of transmission line, short, etc) changing the electromagnetic drag in generators, and, therefore, the speed of rotation of turbines. When that happens, the plant trips of goes off-line to protect the entire grid. Some difference in phase is tolerable, the resulting cross-currents generate heat in the trasmission lines and transformers. It is not sufficient to disconnect a generator from the grid. Since water gates or steam supply can not be closed off fast, the unloaded turbine would accelerate to the point of very violent self-destruction. So the generators are connected to the resistive load to dump the energy there. Those resistors are huge, and go red-hot in seconds. If a gate or valve gets stuck, they melt down, with the resulting explosion of the turbine. You lose some generating capacity but you dont fry everything on the network either. Well... not that simple. A plant going off-line causes sudden load redistribution in the network, potentially causing overload and phase shifting in other plants, etc. A cascading failure, in other words. --vadim
Re: Cascading Failures Could Crash the Global Internet
On Thu, 6 Feb 2003, N. Richard Solis wrote: There are some states that operate their own grids. Texas, for example. Didn't know those chairs took that many megawats :) Paul
Re: Cascading Failures Could Crash the Global Internet
Vadim Antonov wrote: On Thu, 6 Feb 2003, N. Richard Solis wrote: The main cause of AC disruption is a power plant getting out of phase with the rest of the power plants on the grid. This is typically a result of sudden load change (loss of transmission line, short, etc) changing the electromagnetic drag in generators, and, therefore, the speed of rotation of turbines. When that happens, the plant "trips" of goes off-line to protect the entire grid. Some difference in phase is tolerable, the resulting cross-currents generate heat in the trasmission lines and transformers. It is not sufficient to disconnect a generator from the grid. Since water gates or steam supply can not be closed off fast, the unloaded turbine would accelerate to the point of very violent self-destruction. So the generators are connected to the resistive load to dump the energy there. Those resistors are huge, and go red-hot in seconds. If a gate or valve gets stuck, they melt down, with the resulting explosion of the turbine. You lose some generating capacity but you dont fry everything on the network either. Well... not that simple. A plant going off-line causes sudden load redistribution in the network, potentially causing overload and phase shifting in other plants, etc. A cascading failure, in other words. Yeah yeah yeah. I know that everything isn't simple. I actually worked at a power plant so none of this is new to me. Can cascading failures occur? Yes. Witness the Great Blackout in NYC. My point was that there are places where the electrical network is designed to "blow the bolts" to TRY and protect everything. Does it work? Most of the time, yes. All of the time? NO. It is a complicated problem but you'd be suprised at how fast things can happen when you HAVE to keep the system running. There is a tremendous amount of skill concentrated in that field and they do a good job of keeping everything running well. How many turbine overspeed events do you get notified about? Those guys can do a rapid shutdown of a plant VERY quickly. Turning it back on though is a whole different matter. We needed to have one station operating so that we could actually get the big one going. Then we'd take the small one offline and bring it back up quickly to handle specific load peaks. The loss of a single transmission line isn't going to cause a whole station to trip. If you're losing a bunch though, you've probably got lots of other problems to worry about. --vadim
