The recent thread about the Level3 outage ( the posts on the outage itself not the ongoing discussion ) once again showed that Outage threads are a problem area on the NANOG list.
Around 20 posts were sent over the first 2 hours, here is a summary of the content: * is anyone having issues with Level3? * What country, location, where you fed from?? * See [ website ] , I can't get to most web sites if I go via Level3 * Yes. We just experienced an outage in Philadelphia. * theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) * still nothing from Chicago to Colorado, and way too many other places to list. Anyone have a ticket number? * I have a ticket in with L3, their NOC is pretty overwhelmed with phone calls. * [ resent copy of French home connection email ] * seeing that now here from Toronto ON * Same issue here from Chicago and Montreal. Seems anything routing through Washington.Level3 is going to null [ traceroutes ] * According to L3, this issue should be fixed * Here in Richmond Virginia, everything seems to be back to normal now. * Confirmed here as well. * Seems to be normalizing here in Colorado as well * Looks like most providers here in the east coast are routing through level3 again * Any word on the actual cause of the issue? * From what I heard, it was some some malfunction with a router in Washington D.C. * We connect to level3 in Ashburn/DC and saw traffic drop 50% in both directions * We saw our bandwidth drop on our Level3 OC-48 to about half * [ website ] shows everything is back to normal Remember that each of those emails went to 10,000 subscribers on the list, the majority of whom would not have read them till at least the next day. My feeling is that the main NANOG list is not the best forum for people to solicit and diagnose ongoing outages. The threads all seem to be the same, an initial post asking if anyone is seeing something, half a dozen confirmations ( or people saying they don't see anything ) and then a little while later a "it's fixed" and a bunch of confirmations. The list suffers the problems of: - multi-minute turn around - 20 emails per outage being fairly high impact on readers - many readers uninterested in the outage ( at least by the time they read the emails ) The outages mailing list is one resource that has been setup to centralise these sort of outages however there may be other options. eg a quick check of twitter http://search.twitter.com/search?q=level3 shows that many people were micro-blogging about the outage as it happened so possibly twitter or a similar tool might be a better way to handle the majority of these threads ( with perhaps summaries being posted or people just subscribing to the outages mailing list). Thoughts? -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. _______________________________________________ Nanog-futures mailing list [email protected] http://mailman.nanog.org/mailman/listinfo/nanog-futures
