Appears to have been resolved after seven hours. My ticket just says: "We isolated the routing issue and resolved it. The issue was due to a misconfiguration on one our core routers."
Now that the issue is corrected, interestingly enough, my trace goes from Tampa to Chicago instead of Dallas. David -----Original Message----- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Cool Hand Luke Sent: Wednesday, November 19, 2014 11:23 AM To: nanog@nanog.org Subject: Re: level3 issue in chicago (was: FYI, Level 3 issues in Dallas) like david below, we've been getting reports this morning from customers unable to reach various web sites. investigating a number of these reports, the one commonality is level3 in chicago. sites are reachable from level3/cincinnati but not level3/chicago. traceroutes make it one hop past our bgp peer and seem to die there. for one particular remote web server (node1104.follett.com, and possibly others) traceroutes and pings from level3's looking glass (using chicago as the site) complete successfully, while neither does when ran from my gear through the same site. On 11/19/2014 10:46 AM, ga...@completeweb.net wrote: > Also peering problems in Chicago, had to drop BGP to them until they figure it out. > Reported it 4 hours ago, no good response yet. > Gary > >> We have some customers unable to access their websites, seeing this >> on the way to them: >> >> 4 ae-0-11.bar2.Tampa1.Level3.net (4.69.137.110) 0.343 ms 0.423 >> ms >> 0.406 ms >> 5 ae-12-12.ebr1.Dallas1.Level3.net (4.69.137.118) 23.940 ms >> 23.470 ms 23.426 ms >> 6 ae-71-71.csw2.Dallas1.Level3.net (4.69.151.137) 24.026 ms >> ae-81-81.csw3.Dallas1.Level3.net (4.69.151.149) 24.007 ms >> ae-61-61.csw1.Dallas1.Level3.net (4.69.151.125) 23.855 ms >> 7 ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.361 ms >> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.329 ms >> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.936 ms >> 8 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.378 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.387 ms >> vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.963 ms >> 9 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.357 ms >> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.779 ms >> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 62.640 ms 10 >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 24.179 ms 23.297 ms >> 23.403 ms >> 11 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 65.617 ms >> ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.286 ms >> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 23.553 ms >> 12 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.303 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.027 ms 24.028 ms >> 13 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 24.018 ms >> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.374 ms >> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.342 ms >> 14 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.068 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.361 ms 23.348 ms >> 15 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.023 ms >> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.740 ms >> ae-4-90.edge10.Dallas1.Level3.net (4.69.145.210) 23.963 ms >> 16 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.420 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.400 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.317 ms >> 17 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.985 ms >> ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.476 ms >> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 51.762 ms >> 18 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.416 ms >> vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.062 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.051 ms >> 19 ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 44.798 ms >> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.856 ms >> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 22.891 ms 20 >> vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.433 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.333 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.367 ms >> 21 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.475 ms >> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 23.476 ms >> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 64.753 ms >> 22 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 24.008 ms 24.043 >> ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.508 ms >> 23 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.505 ms >> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.958 ms >> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.058 ms >> 24 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.429 ms >> vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.623 ms 25.547 ms >> 25 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.491 ms >> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.551 ms >> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.078 ms >> 26 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.472 ms 24.144 >> ms 24.080 ms >> 27 ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 28.121 ms >> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.848 ms >> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.090 ms >> 28 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.509 ms 24.056 >> ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.522 ms >> 29 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 24.161 ms 24.153 >> ms ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.130 ms 30 >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.606 ms 23.482 ms >> vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.580 ms >> > >
