Didnt know what was a list. On Wed, Jun 10, 2020 at 9:45 AM Andy Ringsmuth via Outages < outages@outages.org> wrote:
> Could we move further discussion to outages-discuss please? > > ---- > Andy Ringsmuth > 5609 Harding Drive > Lincoln, NE 68521-5831 > (402) 304-0083 > a...@andyring.com > > “Better even die free, than to live slaves.” - Frederick Douglas, 1863 > > > On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages < > outages@outages.org> wrote: > > > > Someone didn't have proper protections in place it looks like. > > > > On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages < > outages@outages.org> wrote: > > So another BGP highjacking... > > > > - Mike Bolitho > > > > On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages < > outages@outages.org> wrote: > > Found this on Reddit as an initial RFO, so keep in mind the source > (although r/sysadmin is usually pretty good): > > > > > > > > “A 3rd party network provider was advertising routes which resulted in > our WorldWide traffic becoming severely impeded. This led to IBM Cloud > clients being unable to log-in to their accounts, greatly limited > internet/DC connectivity and other significant network route related > impacts. Network Specialists have made adjustments to route policies to > restore network access, and alleviate the impacts. The overall incident > lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed > Customer Incident Report/Root Cause Analysis as soon as possible” > > > > > > > > Greg Dickinson > > > > Network Engineer > > > > > > > > > > > > 234 Goodwin Crest Drive, Suite 500 > > > > Birmingham, AL 35209 > > > > Office: 205.917.2407 | Mobile: 205.234.6427 > > > > Fax: 205.945.0515 | > > > > greg.dickin...@bryantbank.com | BryantBank.com > > > > Unbeatable Service. Legendary Results. > > > > Facebook | Twitter > > > > > > > > From: Outages <outages-boun...@outages.org> On Behalf Of Jason Kuehl > via Outages > > Sent: Tuesday, June 9, 2020 8:38 PM > > To: James Brown <jbr...@easypost.com> > > Cc: outages@outages.org > > Subject: Re: [outages] IBM Cloud / NetworkLayer Outage > > > > > > > > > > > > This Message originates from outside Bryant Bank. Please use caution > when opening this correspondence, attachments or hyperlinks (URLs). If you > have questions, please contact IT Support. Thank you. > > > > I can't wait for this post mortem. > > > > > > > > On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages < > outages@outages.org> wrote: > > > > As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look > normal to me. > > > > > > > > On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbr...@easypost.com> wrote: > > > > Just wanted to share updates: > > > > > > > > I got through to someone at IBM who confirmed the level of chaos over > there. It looks like they've started deploying some mitigations because I'm > seeing packet loss drop off but latency is now awful for all US locations. > It looks like IBM has failed over all of their peering to go through their > European facilities because every return packet from every POP is coming > from lon03.networklayer.com or ams02.networklayer.com (via Telia). > > > > > > > > On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbr...@easypost.com> wrote: > > > > It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global > outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / > CenturyLink, and a bunch of other providers. Has anyone been able to get > more details? > > > > > > > > -- > > > > James Brown > > > > Network Engineer > > > > > > > > > > > > -- > > > > James Brown > > > > Engineer > > > > > > > > > > > > -- > > > > James Brown > > > > Engineer > > > > _______________________________________________ > > Outages mailing list > > Outages@outages.org > > https://puck.nether.net/mailman/listinfo/outages > > > > > > > > > > > > -- > > > > Sincerely, > > > > Jason W Kuehl > > Cell 920-419-8983 > > jason.w.ku...@gmail.com > > > > > > > > NOTICE: This electronic mail message and any files transmitted with it > are intended exclusively for the individual or entity to which it is > addressed. The message, together with any attachment, may contain > confidential and/or privileged information. Any unauthorized review, use, > print, save, copy, disclosure or distribution is strictly prohibited. If > you have received this message in error, please immediately advise the > sender by reply email and delete copies. Thank you. > > _______________________________________________ > > Outages mailing list > > Outages@outages.org > > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > > Outages mailing list > > Outages@outages.org > > https://puck.nether.net/mailman/listinfo/outages > > > > > > -- > > Sincerely, > > > > Jason W Kuehl > > Cell 920-419-8983 > > jason.w.ku...@gmail.com > > _______________________________________________ > > Outages mailing list > > Outages@outages.org > > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.ku...@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages