Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. http://news.cnet.com/8301-30686_3-20119163-266/blackberry-service-issues-spread-to-u.s-and-canada/ Andrew From: Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Tuesday, October 11, 2011 7:32 PM Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) And continues: “RIM'S SERVICE OUTAGE CONTINUES INTO DAY 2” http://www.channelstv.com/global/news_details.php?nid=29652cat=Politics Frank From:andrew.wallace [mailto:andrew.wall...@rocketmail.com] Sent: Monday, October 10, 2011 2:52 PM To: frnk...@iname.com Cc: outa...@outages.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) RIM shares down as BlackBerry outage continues http://www.marketwatch.com/story/rim-shares-down-as-blackberry-outage-continues-2011-10-10 Andrew From:Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Monday, October 10, 2011 2:47 PM Subject: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) http://english.ahram.org.eg/NewsContent/3/12/23792/Business/Economy/Blackber ry-services-down-worldwide,-Egypt-affected.aspx FYI ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
What kills me is what they have told the public. The lost a core switch. I don't know if they actually mean network switch or not but I'm pretty sure any of us that work on an enterprise environment know how to factor N+1 just for these types of days. And then the backup solution failed? I'm not buying it either. -Hammer- I was a normal American nerd -Jack Herer On 10/12/2011 09:47 AM, andrew.wallace wrote: Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. http://news.cnet.com/8301-30686_3-20119163-266/blackberry-service-issues-spread-to-u.s-and-canada/ Andrew From: Frank Bulkfrnk...@iname.com To: outa...@outages.org Sent: Tuesday, October 11, 2011 7:32 PM Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) And continues: “RIM'S SERVICE OUTAGE CONTINUES INTO DAY 2” http://www.channelstv.com/global/news_details.php?nid=29652cat=Politics Frank From:andrew.wallace [mailto:andrew.wall...@rocketmail.com] Sent: Monday, October 10, 2011 2:52 PM To: frnk...@iname.com Cc: outa...@outages.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) RIM shares down as BlackBerry outage continues http://www.marketwatch.com/story/rim-shares-down-as-blackberry-outage-continues-2011-10-10 Andrew From:Frank Bulkfrnk...@iname.com To: outa...@outages.org Sent: Monday, October 10, 2011 2:47 PM Subject: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) http://english.ahram.org.eg/NewsContent/3/12/23792/Business/Economy/Blackber ry-services-down-worldwide,-Egypt-affected.aspx FYI ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
Never put down to malice which can be more easily explained by stupidity.. or in this case failure. RIM explained the problem earlier.. The messaging and browsing delays being experienced by BlackBerry users in Europe, the Middle East, Africa, India, Brazil, Chile and Argentina were caused by a core switch failure within RIM's infrastructure. Although the system is designed to failover to a back-up switch, the failover did not function as previously tested. As a result, a large backlog of data was generated and we are now working to clear that backlog and restore normal service as quickly as possible. We apologise for any inconvenience and we will continue to keep you informed. This appears to have been a result of a change on monday The problems began at about 11am on Monday. The Guardian understands that RIM was attempting a software upgrade on its database but suffered corruption problems, and that attempts to switch back to an older version led to a collapse http://www.guardian.co.uk/technology/2011/oct/12/blackberry-outage-executive-apologies?newsfeed=true thanks andrew On Wed, Oct 12, 2011 at 3:47 PM, andrew.wallace andrew.wall...@rocketmail.com wrote: Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. http://news.cnet.com/8301-30686_3-20119163-266/blackberry-service-issues-spread-to-u.s-and-canada/ Andrew From: Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Tuesday, October 11, 2011 7:32 PM Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) And continues: “RIM'S SERVICE OUTAGE CONTINUES INTO DAY 2” http://www.channelstv.com/global/news_details.php?nid=29652cat=Politics Frank From:andrew.wallace [mailto:andrew.wall...@rocketmail.com] Sent: Monday, October 10, 2011 2:52 PM To: frnk...@iname.com Cc: outa...@outages.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) RIM shares down as BlackBerry outage continues http://www.marketwatch.com/story/rim-shares-down-as-blackberry-outage-continues-2011-10-10 Andrew From:Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Monday, October 10, 2011 2:47 PM Subject: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) http://english.ahram.org.eg/NewsContent/3/12/23792/Business/Economy/Blackber ry-services-down-worldwide,-Egypt-affected.aspx FYI ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages
RE: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
Maybe they use the same security solutions as Playstation Network does... that would explain a lot suddenly. Paul -Original Message- From: andrew.wallace [mailto:andrew.wall...@rocketmail.com] Sent: Wednesday, October 12, 2011 10:47 AM To: frnk...@iname.com Cc: nanog@nanog.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. http://news.cnet.com/8301-30686_3-20119163-266/blackberry-service-issues-spread-to-u.s-and-canada/ Andrew From: Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Tuesday, October 11, 2011 7:32 PM Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) And continues: “RIM'S SERVICE OUTAGE CONTINUES INTO DAY 2” http://www.channelstv.com/global/news_details.php?nid=29652cat=Politics Frank From:andrew.wallace [mailto:andrew.wall...@rocketmail.com] Sent: Monday, October 10, 2011 2:52 PM To: frnk...@iname.com Cc: outa...@outages.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) RIM shares down as BlackBerry outage continues http://www.marketwatch.com/story/rim-shares-down-as-blackberry-outage-continues-2011-10-10 Andrew From:Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Monday, October 10, 2011 2:47 PM Subject: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) http://english.ahram.org.eg/NewsContent/3/12/23792/Business/Economy/Blackber ry-services-down-worldwide,-Egypt-affected.aspx FYI ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
On 10/12/11 07:47 , andrew.wallace wrote: Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. http://news.cnet.com/8301-30686_3-20119163-266/blackberry-service-issues-spread-to-u.s-and-canada/ North American outages of the blackberry platform (particularly related to upgrades gone wrong) were not uncommon. Think for example sept 10, dec 18 and dec 22 2009. Andrew From: Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Tuesday, October 11, 2011 7:32 PM Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) And continues: “RIM'S SERVICE OUTAGE CONTINUES INTO DAY 2” http://www.channelstv.com/global/news_details.php?nid=29652cat=Politics Frank From:andrew.wallace [mailto:andrew.wall...@rocketmail.com] Sent: Monday, October 10, 2011 2:52 PM To: frnk...@iname.com Cc: outa...@outages.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) RIM shares down as BlackBerry outage continues http://www.marketwatch.com/story/rim-shares-down-as-blackberry-outage-continues-2011-10-10 Andrew From:Frank Bulk frnk...@iname.com To: outa...@outages.org Sent: Monday, October 10, 2011 2:47 PM Subject: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) http://english.ahram.org.eg/NewsContent/3/12/23792/Business/Economy/Blackber ry-services-down-worldwide,-Egypt-affected.aspx FYI ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
On Wed, 12 Oct 2011 07:47:13 PDT, andrew.wallace said: Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. It ain't sabotage till you rule out misconfigured router. Consider the actual real-world threat models and their likelyhoods: 1) Insufficiently caffienated network engineer - this *NEVER* happens in real life, it's a total Bruce Schneier caliber movie-plot scenario. 2) Somebody sabotaging a RIM router. This is more likely, because there's just *bazillions* of people out there that stand to benefit from a RIM outage (and in fact profit more from an outage than from being able to watch traffic as it goes by). It's just a question of which one of those bazillions did it *this* time. Andrew, you *really* need to learn what the actual failure modes and root causes in real-life production networks are, and draw conclusions from reality, not whatever MI-7 inspired dream world the claim of sabotage came from. pgpBxmKLHmFJ6.pgp Description: PGP signature
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
On Wed, 12 Oct 2011 09:52:02 CDT, -Hammer- said: What kills me is what they have told the public. The lost a core switch. I don't know if they actually mean network switch or not but I'm pretty sure any of us that work on an enterprise environment know how to factor N+1 just for these types of days. And then the backup solution failed? I'm not buying it either. Yeah, and that extra comma in the one config file that didn't make a difference when you tested the failover in the lab *never* makes a difference when it hits in the production network, right? Or they changed the config of the primary and it didn't get propogated just right to the backup, or they had mismatched firmware levels on blades in the blades on the primary and backup switches, so traffic that didn't tickle a bug on the primary blades caused the blade to crash on the backup, or... Anybody on this list who's been around long enough probably has enough We should have had N+2 because the N+1'th device failed too stories to drain *several* pitchers of beer at a good pub... I've even had one case where my butt got *saved* from a ohnosecond-class whoops because the N+1'th device *was* crashed (stomped a config file, it replicated, was able to salvage a copy from a device that didn't replicate because it was down at the time). pgpP55SUQUVfz.pgp Description: PGP signature
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
+1 On Oct 12, 2011 11:51 AM, valdis.kletni...@vt.edu wrote: On Wed, 12 Oct 2011 09:52:02 CDT, -Hammer- said: What kills me is what they have told the public. The lost a core switch. I don't know if they actually mean network switch or not but I'm pretty sure any of us that work on an enterprise environment know how to factor N+1 just for these types of days. And then the backup solution failed? I'm not buying it either. Yeah, and that extra comma in the one config file that didn't make a difference when you tested the failover in the lab *never* makes a difference when it hits in the production network, right? Or they changed the config of the primary and it didn't get propogated just right to the backup, or they had mismatched firmware levels on blades in the blades on the primary and backup switches, so traffic that didn't tickle a bug on the primary blades caused the blade to crash on the backup, or... Anybody on this list who's been around long enough probably has enough We should have had N+2 because the N+1'th device failed too stories to drain *several* pitchers of beer at a good pub... I've even had one case where my butt got *saved* from a ohnosecond-class whoops because the N+1'th device *was* crashed (stomped a config file, it replicated, was able to salvage a copy from a device that didn't replicate because it was down at the time).
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
Idiotberry Envoyé de mon iPhone Le 12 oct. 2011 à 17:55, Charles Mills w3y...@gmail.com a écrit : +1 On Oct 12, 2011 11:51 AM, valdis.kletni...@vt.edu wrote: On Wed, 12 Oct 2011 09:52:02 CDT, -Hammer- said: What kills me is what they have told the public. The lost a core switch. I don't know if they actually mean network switch or not but I'm pretty sure any of us that work on an enterprise environment know how to factor N+1 just for these types of days. And then the backup solution failed? I'm not buying it either. Yeah, and that extra comma in the one config file that didn't make a difference when you tested the failover in the lab *never* makes a difference when it hits in the production network, right? Or they changed the config of the primary and it didn't get propogated just right to the backup, or they had mismatched firmware levels on blades in the blades on the primary and backup switches, so traffic that didn't tickle a bug on the primary blades caused the blade to crash on the backup, or... Anybody on this list who's been around long enough probably has enough We should have had N+2 because the N+1'th device failed too stories to drain *several* pitchers of beer at a good pub... I've even had one case where my butt got *saved* from a ohnosecond-class whoops because the N+1'th device *was* crashed (stomped a config file, it replicated, was able to salvage a copy from a device that didn't replicate because it was down at the time).
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
I think it raises serious questions about RIM's DR strategy if a DB corruption or switch failure or whatever can cause this much outage. 'Surely' RIM have an second site that is independent of the primary (within reason) that they could of flipped to when they realised the DB was borked. If not then any business that relies on them needs to be shouting from the rooftops to get RIM to fix it. Chris. On 12 Oct 2011, at 16:49, valdis.kletni...@vt.edu wrote: On Wed, 12 Oct 2011 09:52:02 CDT, -Hammer- said: What kills me is what they have told the public. The lost a core switch. I don't know if they actually mean network switch or not but I'm pretty sure any of us that work on an enterprise environment know how to factor N+1 just for these types of days. And then the backup solution failed? I'm not buying it either. Yeah, and that extra comma in the one config file that didn't make a difference when you tested the failover in the lab *never* makes a difference when it hits in the production network, right? Or they changed the config of the primary and it didn't get propogated just right to the backup, or they had mismatched firmware levels on blades in the blades on the primary and backup switches, so traffic that didn't tickle a bug on the primary blades caused the blade to crash on the backup, or... Anybody on this list who's been around long enough probably has enough We should have had N+2 because the N+1'th device failed too stories to drain *several* pitchers of beer at a good pub... I've even had one case where my butt got *saved* from a ohnosecond-class whoops because the N+1'th device *was* crashed (stomped a config file, it replicated, was able to salvage a copy from a device that didn't replicate because it was down at the time).
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
I have been witness to N+1 HUMAN failures but never a N+1 hardware failure or system/design failure that warranted questioning the need for N+2. Usually your N+1 failure is (as already referenced) pasting in a bad config that gets replicated or something like that. Not saying the hardware is perfect. It's just that I haven't personally seen a full blown failure like that without human help. Closest example would be an update that wasn't properly vetted in dev/test before migrating to prod. I've seen a few of those that I guess you could blame on the system. Even though the humans could have tested better -Hammer- I was a normal American nerd -Jack Herer On 10/12/2011 10:58 AM, Chris Campbell wrote: I think it raises serious questions about RIM's DR strategy if a DB corruption or switch failure or whatever can cause this much outage. 'Surely' RIM have an second site that is independent of the primary (within reason) that they could of flipped to when they realised the DB was borked. If not then any business that relies on them needs to be shouting from the rooftops to get RIM to fix it. Chris. On 12 Oct 2011, at 16:49, valdis.kletni...@vt.edu wrote: On Wed, 12 Oct 2011 09:52:02 CDT, -Hammer- said: What kills me is what they have told the public. The lost a core switch. I don't know if they actually mean network switch or not but I'm pretty sure any of us that work on an enterprise environment know how to factor N+1 just for these types of days. And then the backup solution failed? I'm not buying it either. Yeah, and that extra comma in the one config file that didn't make a difference when you tested the failover in the lab *never* makes a difference when it hits in the production network, right? Or they changed the config of the primary and it didn't get propogated just right to the backup, or they had mismatched firmware levels on blades in the blades on the primary and backup switches, so traffic that didn't tickle a bug on the primary blades caused the blade to crash on the backup, or... Anybody on this list who's been around long enough probably has enough We should have had N+2 because the N+1'th device failed too stories to drain *several* pitchers of beer at a good pub... I've even had one case where my butt got *saved* from a ohnosecond-class whoops because the N+1'th device *was* crashed (stomped a config file, it replicated, was able to salvage a copy from a device that didn't replicate because it was down at the time).
RE: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
-Original Message- From: -Hammer- [mailto:bhmc...@gmail.com] Sent: 12 October 2011 17:10 To: nanog@nanog.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) I have been witness to N+1 HUMAN failures but never a N+1 hardware failure or system/design failure that warranted questioning the need for N+2. Usually your N+1 failure is (as already referenced) pasting in a bad config that gets replicated or something like that. Not saying the hardware is perfect. It's just that I haven't personally seen a full blown failure like that without human help. You have not seen VIP2-40s and CEF in action ;-) -- Leigh Porter __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
I have and totally get the point ... -- Michael Gatti cell.949.735.5612 ekim.it...@gmail.com (UTC-8) On Oct 12, 2011, at 9:12 AM, Leigh Porter wrote: -Original Message- From: -Hammer- [mailto:bhmc...@gmail.com] Sent: 12 October 2011 17:10 To: nanog@nanog.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) I have been witness to N+1 HUMAN failures but never a N+1 hardware failure or system/design failure that warranted questioning the need for N+2. Usually your N+1 failure is (as already referenced) pasting in a bad config that gets replicated or something like that. Not saying the hardware is perfect. It's just that I haven't personally seen a full blown failure like that without human help. You have not seen VIP2-40s and CEF in action ;-) -- Leigh Porter __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
RE: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
They are out there scrambling, trying to figure out where the truck that hit them came from. The PIO has been told to make up a story. Ralph Brandt Communications Engineer HP Enterprise Services Telephone +1 717.506.0802 FAX +1 717.506.4358 Email ralph.bra...@pateam.com 5095 Ritter Rd Mechanicsburg PA 17055 -Original Message- From: -Hammer- [mailto:bhmc...@gmail.com] Sent: Wednesday, October 12, 2011 10:52 AM To: nanog@nanog.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) What kills me is what they have told the public. The lost a core switch. I don't know if they actually mean network switch or not but I'm pretty sure any of us that work on an enterprise environment know how to factor N+1 just for these types of days. And then the backup solution failed? I'm not buying it either. -Hammer- I was a normal American nerd -Jack Herer On 10/12/2011 09:47 AM, andrew.wallace wrote: Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. http://news.cnet.com/8301-30686_3-20119163-266/blackberry-service-issues-spread-to-u.s-and-canada/ Andrew From: Frank Bulkfrnk...@iname.com To: outa...@outages.org Sent: Tuesday, October 11, 2011 7:32 PM Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) And continues: “RIM'S SERVICE OUTAGE CONTINUES INTO DAY 2” http://www.channelstv.com/global/news_details.php?nid=29652cat=Politics Frank From:andrew.wallace [mailto:andrew.wall...@rocketmail.com] Sent: Monday, October 10, 2011 2:52 PM To: frnk...@iname.com Cc: outa...@outages.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) RIM shares down as BlackBerry outage continues http://www.marketwatch.com/story/rim-shares-down-as-blackberry-outage-continues-2011-10-10 Andrew From:Frank Bulkfrnk...@iname.com To: outa...@outages.org Sent: Monday, October 10, 2011 2:47 PM Subject: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) http://english.ahram.org.eg/NewsContent/3/12/23792/Business/Economy/Blackber ry-services-down-worldwide,-Egypt-affected.aspx FYI ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list outa...@outages.org https://puck.nether.net/mailman/listinfo/outages __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
- Original Message - From: Valdis Kletnieks valdis.kletni...@vt.edu On Wed, 12 Oct 2011 07:47:13 PDT, andrew.wallace said: Guys the outage has moved to U.S and Canada, I think we need to look at this perhaps being sabotage. It ain't sabotage till you rule out misconfigured router. Andrew, you *really* need to learn what the actual failure modes and root causes in real-life production networks are, and draw conclusions from reality, not whatever MI-7 inspired dream world the claim of sabotage came from. In fairness, Valdis, Andrew did not say this was obviously sabotage. He suggested that that possibility be added to the list of things which the RIM employees tasked with finding a root cause consider. I think the old filtering rule applies here: Once is happenstance. Twice is coincidence. Three times is enemy action. If this turns out to look like it came from 3 or more non-cascading failures, then sabotage will look a little more likely. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.)
Again. I know those stories are out there. I'm blessed with a lower profile or higher karma. One of the two. digging thru cube to fine wood to knock on -Hammer- I was a normal American nerd -Jack Herer On 10/12/2011 11:53 AM, Mike Gatti wrote: I have and totally get the point ... -- Michael Gatti cell.949.735.5612 ekim.it...@gmail.com (UTC-8) On Oct 12, 2011, at 9:12 AM, Leigh Porter wrote: -Original Message- From: -Hammer- [mailto:bhmc...@gmail.com] Sent: 12 October 2011 17:10 To: nanog@nanog.org Subject: Re: [outages] News item: Blackberry services down worldwide, Egypt affected (not N.A.) I have been witness to N+1 HUMAN failures but never a N+1 hardware failure or system/design failure that warranted questioning the need for N+2. Usually your N+1 failure is (as already referenced) pasting in a bad config that gets replicated or something like that. Not saying the hardware is perfect. It's just that I haven't personally seen a full blown failure like that without human help. You have not seen VIP2-40s and CEF in action ;-) -- Leigh Porter __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
