I'm seeing traffic outbound from Level 3 Tampa going to Frankfurt Germany currently, so they're definitely doing something wrong.
3 xe-10-3-1.bar1.Tampa1.Level3.net (4.53.172.29) 13.642 ms 13.650 ms 13.645 ms 4 ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.129 ms 117.100 ms ae-2-70.edge3.Frankfurt1.Level3.net (4.69.154.71) 116.928 ms 5 ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.724 ms ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.079 ms ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.756 ms > -----Original Message----- > From: Outages [mailto:outages-boun...@outages.org] On Behalf > Of Matthew Topper via Outages > Sent: Thursday, August 27, 2015 2:29 PM > To: outages@outages.org > Subject: Re: [outages] Level3/TW Telecom issues? > > Seeing on and off in upstate NY > > > Matthew Topper > > -----Original Message----- > From: Outages [mailto:outages-boun...@outages.org] On Behalf > Of Jason Hellenthal via Outages > Sent: Thursday, August 27, 2015 2:23 PM > To: outages@outages.org > Subject: Re: [outages] Level3/TW Telecom issues? > > Retracting my statement from yesterday this is still a solid > issue here. Yes absolutely still continuing. Root Cause would > be a great explanation at this point. > > > On Aug 27, 2015, at 13:10, Ryan Brooks via Outages > <outages@outages.org> wrote: > > > > 24 hours of on and off packet loss, more than just Chicago. > More communication would be helpful. NOC can't seem to > decide if it's transport or routing. Any insights? > > > >> On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages > <outages@outages.org> wrote: > >> > >> We just got another update saying they still have issues. > >> > >> > >> IP services are affected across multiple markets. The IP > NOC and IP and Transport Tier III are actively > troubleshooting the issue. Operations Engineering is engaged > and Field Services is on site in Chicago, IL investigating the issue. > >> tw telecom has identified a network failure affecting your > services. tw telecom tier-2 group is investigating the issue > at this time. > >> > >> Thanks, > >> Bobin > >> > >> From: Outages [mailto:outages-boun...@outages.org] On > Behalf Of Aaron > >> Gross via Outages > >> Sent: Thursday, August 27, 2015 10:55 AM > >> To: Bob Hancock > >> Cc: outages@outages.org > >> Subject: Re: [outages] Level3/TW Telecom issues? > >> > >> We are seeing the same things that happened yesterday on > our circuits in Milwaukee and Chicago areas. Lots of packet > loss and large latency through Chicago exchanges. > >> > >> On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages > <outages@outages.org> wrote: > >> We were seeing substantial loss through Chicago this > morning but it > >> seems to have subsided at this time. > >> > >> > >> Bob Hancock > >> > >> > >> On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote: > >> > we are still seeing significant packet loss to 3 of our sites in > >> > the Midwest. > >> > > >> > On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages > >> > <outages@outages.org <mailto:outages@outages.org>> wrote: > >> > > >> > We got this from Level3____ > >> > > >> > __ __ > >> > > >> > Hello, > >> > > >> > The network outage has been resolved. > >> > > >> > *Event Conclusion Summary* > >> > > >> > Start: August 26, 2015 19:15 GMT > >> > Stop: August 27, 2015 00:14 GMT > >> > > >> > Root Cause: An unknown issue in Chicago, IL was > impacting IP services. > >> > Fix Action: Adjustments were made to resolve the > issue and restore > >> > services. > >> > > >> > Summary: > >> > The IP NOC responded to an unknown issue in Chicago, > IL that was > >> > impacting IP services. Tier III Technical Support > were engaged to > >> > assist with investigations, and adjustments were > made to resolve the > >> > issue and stabilize traffic. Monitoring continues to ensure > >> > stability. Should further issues be experienced, > please contact the > >> > Level 3 Technical Service Center. > >> > > >> > > >> > ____ > >> > > >> > __ __ > >> > > >> > Thanks,____ > >> > > >> > Bobin____ > >> > > >> > __ __ > >> > > >> > *From:*Outages [mailto:outages-boun...@outages.org > >> > <mailto:outages-boun...@outages.org>] *On Behalf Of > *Ryan Gelobter > >> > via Outages > >> > *Sent:* Thursday, August 27, 2015 9:44 AM > >> > *To:* Rafael Possamai > >> > *Cc:* outages@outages.org <mailto:outages@outages.org> > >> > *Subject:* Re: [outages] Level3/TW Telecom issues?____ > >> > > >> > __ __ > >> > > >> > Was anyone able to get some useful information out of Level3 > >> > regarding this yesterday? Also is anyone else seeing similar > >> > issues?____ > >> > > >> > On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" > >> > <outages@outages.org <mailto:outages@outages.org>> wrote:____ > >> > > >> > Here's a graph showing some packet loss: > >> > http://imgur.com/KVAzetA > >> > > >> > > <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL > >> > > CzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LO > >> > > VJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HF > >> > > kr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQz > >> > h0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____ > >> > > >> > __ __ > >> > > >> > __ __ > >> > > >> > On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages > >> > <outages@outages.org <mailto:outages@outages.org>> wrote:____ > >> > > >> > Anyone else having issues on Level3? We're seeing > packet loss with > >> > anything that traverses Chicago.____ > >> > > >> > __ __ > >> > > >> > Thanks,____ > >> > > >> > Matt____ > >> > > >> > This message and attachment(s), if any, is intended > for the sole use > >> > of the individual and/or entity for which it is > addressed, and may > >> > contain information that is privileged, confidential > and prohibited > >> > from disclosure under applicable law. If you are not > the addressee, > >> > or authorized to receive this on behalf of the > addressee, you are > >> > hereby notified that you may not use, copy, disclose > or distribute > >> > to anyone this message or any part thereof. If you > have received > >> > this in error, please immediately advise the sender > by e-mail and > >> > delete this information and all attachments from > your computer and > >> > network. Thank you. ____ > >> > > >> > > >> > _______________________________________________ > >> > Outages mailing list > >> > Outages@outages.org <mailto:Outages@outages.org> > >> > https://puck.nether.net/mailman/listinfo/outages > >> > > >> > > <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs > >> > > ppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrug > >> > > sotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5 > >> > > bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_j > >> > > w09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH> > >> > ____ > >> > > >> > __ __ > >> > > >> > > >> > _______________________________________________ > >> > Outages mailing list > >> > Outages@outages.org <mailto:Outages@outages.org> > >> > https://puck.nether.net/mailman/listinfo/outages > >> > > >> > > <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV > >> > > EVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYK > >> > > rugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl > >> > > 6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ > >> > > 8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWo > >> > Uq>____ > >> > > >> > > >> > _______________________________________________ > >> > Outages mailing list > >> > Outages@outages.org <mailto:Outages@outages.org> > >> > https://puck.nether.net/mailman/listinfo/outages > >> > > >> > > <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU > >> > > OOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYw > >> > > UMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAa > >> > > mz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D > >> > > 00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4> > >> > > >> > > >> > > >> > > >> > -- > >> > To him who is able to keep you from falling and to present you > >> > before his glorious presence without fault and with great joy > >> > > >> > > >> > _______________________________________________ > >> > Outages mailing list > >> > Outages@outages.org > >> > > http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB > >> > > USztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1Nx > >> > > R1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6 > >> > > h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00 > >> > > CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp > >> > > >> The information contained in this E-mail message is > privileged, confidential, and may be protected from > disclosure; please be aware that any unauthorized use, > printing, copying, disclosure or dissemination of this > communication may be subject to legal restriction or > sanction. If you think that you have received this E-mail > message in error, please reply to the sender. This E-mail > message and any attachments have been scanned for viruses and > are believed to be free of any virus or other defect that > might affect any computer system into which it is received > and opened. It is the responsibility, however, of the > recipient to ensure that it is virus free. > >> _______________________________________________ > >> Outages mailing list > >> Outages@outages.org > >> https://puck.nether.net/mailman/listinfo/outages > >> > >> _______________________________________________ > >> Outages mailing list > >> Outages@outages.org > >> https://puck.nether.net/mailman/listinfo/outages > > > > _______________________________________________ > > Outages mailing list > > Outages@outages.org > > https://puck.nether.net/mailman/listinfo/outages > > > -- > Jason Hellenthal > JJH48-ARIN > > > > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages