It would also be interesting to know which backbone/core product requires a reboot to
activate OSPF configuration changes. Sounds like something one should stay away from.
Pete
Frank Scalzo wrote:
>
> Whoops! 2 hours to find routers w/o an IGP tsk tsk.
>
> Dear AT&T IP Services Customer,
>
> Please be advised of the following:
>
> Date: 8/28/02
> Customer Care TT#: 1070909
> Start of Impairment: 14:56 ET
> End of Impairment: 16:52 ET
>
> On behalf of AT&T, we would like to extend our apologies for any inconveniences to
>your business caused by the
> service impairment and delay in restoring your service.
>
> AT&T IP Services customers with traffic routed through our Chicago backbone routers
>may have been affected by
> this impairment.
>
> Our Network Engineers found that OSPF* network statements were missing from two
>backbone routers, causing the
> routing issues that customers were experiencing. AT&T Network Engineers manually
>reloaded the OSPF network
> statements and then proceeded to reboot the routers in order for the changes to
>take effect.
>
> The network statements were mistakenly deleted during a routine configuration
>update to our backbone routers.
> Management has been made aware of these findings and appropriate actions will be
>taken to ensure this situation
> does not re-occur.
>
> Prior reported information was leading the AT&T Network Engineers to believe that
>there were issues with the route
> reflectors being out of synch. However, this was only symptomatic of the problem
>stated above. In other words, due
> to the missing network statements, the route reflectors were not advertising
>certain networks; therefore traffic was
> not properly routed.
>
> It is AT&T?s goal to provide the highest level of service to our customers.
> We appreciate your business and look forward to continuing our relationship in the
>future.
>
> *Routing protocol (Open Shortest Path First)
>
> Depending on the particular IP access services to which you are subscribed you may
>also find additional information
> as it becomes available at:
>
> AT&T MIS:
> https://mis-att.bus.att.com/
>
> AT&T VPNS:
> http://www.vpn.att.net
>
> If you require further information, please feel free to contact AT&T at:
>
> AT&T MIS: 1-888-613-6330
> AT&T VPNS: 1-888-613-6501
>
> Thank you for using AT&T.
>
> Sincerely,
>
> The AT&T Customer Care Team
>
> -----Original Message-----
> From: Matt Levine [mailto:[EMAIL PROTECTED]]
> Sent: Wed 8/28/2002 6:21 PM
> To: Mike Tancsa
> Cc: Wes Bachman; [EMAIL PROTECTED]
> Subject: Re: AT&T NYC
>
> On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
>
> >
> >
> > route-server.ip.att.net is not currently reachable, but AS15290's
> > router server is for those who want a view on things...
>
> Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody
> else, including AS7018 internal)..
>
> >
> > route-server.east.attcanada.com.
> > and
> > route-server.west.attcanada.com.
> >
> > which come in handy :-)
> >
> > ---Mike
> >
> > At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
> >
> >
> >> I am seeing this as well. One of my upstreams (AT&T Canada- 15290)
> >> has connections with AT&T US (7018) in Chicago and Vancouver.
> >> Chicago seems to have disappeared for me and all traffic bound via
> >> that path is going via Vancouver now.
> >>
> >> ---Mike
> >>
> >> At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
> >>
> >>> Bryan,
> >>>
> >>> There is a known AT&T outage in Chicago currently. Could this be
> >>> effecting you in some way?
> >>>
> >>> -Wes
> >>>
> >>> On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
> >>> >
> >>> > Anyone seeing any problems with ATT in new york?
> >>> >
> >>> >
> >>> > Best regards,
> >>> >
> >>> >
> >>> > Bryan Heitman
> >>> > Interland, Inc.
> >>> --
> >>> Wes Bachman
> >>> System & Network Administration, Software Development
> >>> Leepfrog Technologies, Inc.
> >>> [EMAIL PROTECTED]
> >
> >
> --
> Matt Levine
> @Home: [EMAIL PROTECTED]
> @Work: [EMAIL PROTECTED]
> ICQ : 17080004
> AIM : exile
> GPG : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6C0D04CF
> "The Trouble with doing anything right the first time is that nobody
> appreciates how difficult it was." -BIX