I highly doubt bandwidth or Internet connectivity are the whole story. Anyone who thinks that a private connection to an otherwise public-facing CLEC/ITSP is going to eliminate or substantially reduce their exposure to problems of this nature is thinking naively.
> On Sep 30, 2021, at 8:50 AM, Mike Hammett <voice...@ics-il.net> wrote: > > I know about peering well. > > > Why I ask is that from an IP traffic engineering scenario, PNIs are great. > They really limit the failure domain. However, because there are very few > details coming out, we don't know if it's even a data capacity limitation. > Maybe the SBCs are maxed out in terms of compute capacity and therefore, > bypassing the Internet doesn't actually do anything for you. > > Maybe there's more to it than just a DDoS. > > > > ----- > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > > > Midwest Internet Exchange > http://www.midwest-ix.com > > > > From: "Henning Westerholt" <h...@skalatan.de> > To: "Mike Hammett" <voice...@ics-il.net>, "Jared Geiger" <ja...@compuwizz.net> > Cc: "VoiceOps" <voiceops@voiceops.org> > Sent: Thursday, September 30, 2021 7:44:23 AM > Subject: RE: Bandwidth - Monday Outage > > Maybe they do not want to tell, maybe they are not allowed to tell, maybe > they are not on this mailing list. > > Also, in my (European) experience, its common to have a private peering when > you have a certain volume with your termination/interconnection provider. > > Cheers, > > Henning > > From: VoiceOps <voiceops-boun...@voiceops.org> On Behalf Of Mike Hammett > Sent: Thursday, September 30, 2021 2:35 PM > To: Jared Geiger <ja...@compuwizz.net> > Cc: VoiceOps <voiceops@voiceops.org> > Subject: Re: [VoiceOps] Bandwidth - Monday Outage > > "I assume those companies connect with Bandwidth privately versus public > Internet." > > I've asked on here if anyone is connected to Bandwidth.com via PNI and still > having issues, but I haven't seen anything back. > > > > ----- > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > > > Midwest Internet Exchange > http://www.midwest-ix.com > > > > From: "Jared Geiger" <ja...@compuwizz.net> > To: "VoiceOps" <voiceops@voiceops.org> > Sent: Wednesday, September 29, 2021 11:20:31 PM > Subject: Re: [VoiceOps] Bandwidth - Monday Outage > > Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS > protection. > > Then Cloudflare Magic Transit wasn't so magic today. > https://www.cloudflarestatus.com/incidents/kctplzfbf2j2 > > VoIP needs to decouple from the TDM PSTN legacy so we can get federated and > authenticated ENUM IP peering at IXPs or something similar to what the > GRX/IPX does in the mobile world. I'm sure this exists to some extent between > the big guys already, but us little guys need in on the action to make > services more robust. > > I'm surprised more people haven't complained that Google Voice and Microsoft > Teams numbers aren't working. I've been too busy to test during these outages > to test. I assume those companies connect with Bandwidth privately versus > public Internet. > > On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <mary...@backuptelecom.com> > wrote: > I will just add that I've helped carriers of all types install and > maintain their networks for the last 21 years. I've worked with every > major ILEC and RBOC and the amount of anti-competitive tactics I've > witnessed over the years has always been through the roof because there > was never a motivation for the big guys to share their networks with the > little guys. > > These kind of tactics are nothing new, so I suspect the attack > originated from a domestic player/group of players and Bandwidth will > not be their only target. My suggestion to everyone would be to make > your networks as redundant as possible so you don't have to rely on any > one carrier. Don't burn bridges with any carriers either because you > never know when you might need them again. > > > MARY LOU CAREY > BackUP Telecom Consulting > Office: 615-791-9969 > Cell: 615-796-1111 > > On 2021-09-29 01:39 PM, Mary Lou Carey wrote: > > This smells very fishy to me. The fact that a long-term attack has > > been targeted at one of a few companies that host other carrier's > > services AND provides 911 services the weekend before STIR/SHAKEN's > > implementation takes place does not appear to be a coincidence to me. > > Carriers fight attacks off every day, but In all my years of working > > in the industry, I've never seen an attack last so long that it had > > the potential to take a carrier out of business. In my opinion, this > > wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth > > seems to have a lot of resources and purposely intends to take > > Bandwidth out. Call me crazy if you want, but when I smell fish I'm > > usually not wrong! > > > > MARY LOU CAREY > > BackUP Telecom Consulting > > Office: 615-791-9969 > > Cell: 615-796-1111 > > > > On 2021-09-29 01:03 PM, Mark Wiles wrote: > >> While we all might love to know what they’ve done to TRY to mitigate > >> the issue; it’s reasonable to assume that they’d be fairly quiet > >> about what they’re doing/trying to do. Right now, I’d rather them > >> keep a low profile and simply get the issue addressed. You know > >> they’re hemorrhaging customers left-and-right due to port-aways. > >> > >> From: VoiceOps <voiceops-boun...@voiceops.org> On Behalf Of Ryan > >> Delgrosso > >> Sent: Wednesday, September 29, 2021 1:52 PM > >> To: voiceops@voiceops.org > >> Subject: Re: [VoiceOps] Bandwidth - Monday Outage > >> > >> FYI a pretty weak but publicly referencable acknowledgement of whats > >> going on > >> > >> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ > >> [4] > >> > >> On 9/29/2021 10:37 AM, Pete Eisengrein wrote: > >> > >>> They have publicly acknowledge it as a DDoS ( > >>> > >> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ > >>> [1] ) , but being pretty tight-lipped with specifics on what it is > >>> or how they are mitigating. > >>> > >>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez > >>> <caalva...@gmail.com> wrote: > >>> > >>> Is this some sort of ransom event against them maybe? And what are > >>> the rest of you telling your customers? We seem to have only a few > >>> specifically complaining, but those are complaining a lot. > >>> > >>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic > >>> <ivan.kovace...@startelecom.ca> wrote: > >>> > >>> Happening again. > >>> > >>> https://status.bandwidth.com/ [2] > >>> > >>> [3] > >>> > >>> Ivan Kovacevic > >>> _Co-Founder and VP Client Services_ > >>> > >>> [3] > >>> > >>> [3] > >>> > >>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps > >>> <voiceops@voiceops.org> wrote: [3] > >>> > >>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote: > >>> > >>>> Nothing meaningful other than the normal public party line. > >>>> > >>>> I too have heard unofficially that its DDOS, which makes sense > >>> given the > >>>> recurring nature. > >>>> > >>>> 4.5hrs down Sat > >>> > >>> Our monitoring showed 2 hours 47 minutes of actual service > >>> affecting > >>> outages across Voice (Inbound and Outbound), Messaging, and > >>> API/Portal. > >>> > >>> The issue started at 3pm and recovered at 5:47pm EDT. We reported > >>> it to > >>> the TAC at 3:07pm, they did not post on Status until 3:31pm. > >>> > >>>> Some small downtime Sun > >>>> > >>>> Now deep into Monday with problems. > >>>> > >>>> Its not a good look, but id like some more transparency. > >>> > >>> DDoS attacks are real and hard to null route. You've got millions > >>> of IP > >>> addresses slamming you with data. Your router has a capacity, and > >>> your > >>> router cannot handle all of that extra crap data along with all of > >>> our > >>> traffic too. > >>> > >>> I'm sure BW will be investing in some beefy hardware that will be > >>> able to > >>> better handle DDoS attacks, as well as working more closely with > >>> their > >>> peering providers. I have to assume that they were getting > >>> gigabits of > >>> traffic, overwhelming their links in addition to their edge > >>> routers. > >>> > >>> Cloudflare details how they do it here: > >>> > >>> > >> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection > >>> > >>> Not much to be transparent about. The Internet is an unfriendly > >>> place, and > >>> bad actors can rain hell upon any public IP they want. Unsecured > >>> laptops, > >>> desktops, TVs, IOT devices, etc, all contribute just a little tiny > >>> bit, > >>> and all focus on one single point, kinda like those giant solar > >>> farms with > >>> the mirrors and single tower in the middle to boil the molten > >>> salt. > >>> > >>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of > >>> unsecured devices on the Internet. > >>> > >>> > >> --------------------------------------------------------------------------- > >>> Peter Beckman > >>> Internet Guy > >>> beck...@angryox.com > >>> https://www.angryox.com/ > >>> > >> ---------------------------------------------------------------------------_______________________________________________ > >>> VoiceOps mailing list > >>> VoiceOps@voiceops.org > >>> https://puck.nether.net/mailman/listinfo/voiceops > >>> _______________________________________________ > >>> VoiceOps mailing list > >>> VoiceOps@voiceops.org > >>> https://puck.nether.net/mailman/listinfo/voiceops > >>> > >>> [3] > >>> > >>> [3] > >>> > >>> NOTE: This email message and any attachments are for the sole use of > >>> the intended recipient(s) and may contain confidential and/or > >>> privileged information. Any unauthorized review, use, disclosure or > >>> distribution is prohibited. If you are not the intended recipient, > >>> please contact the sender by replying to this email, and destroy all > >>> copies of the original message. [3] > >>> > >>> _______________________________________________ > >>> VoiceOps mailing list > >>> VoiceOps@voiceops.org > >>> https://puck.nether.net/mailman/listinfo/voiceops > >> > >> _______________________________________________ > >> VoiceOps mailing list > >> VoiceOps@voiceops.org > >> https://puck.nether.net/mailman/listinfo/voiceops > >> > >> _______________________________________________ [3] > >> > >> VoiceOps mailing list [3] > >> > >> VoiceOps@voiceops.org [3] > >> > >> https://puck.nether.net/mailman/listinfo/voiceops [3] > >> > >> > >> Links: > >> ------ > >> [1] > >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1 > >> [2] > >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1 > >> [3] > >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1 > >> [4] > >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1 > >> _______________________________________________ > >> VoiceOps mailing list > >> VoiceOps@voiceops.org > >> https://puck.nether.net/mailman/listinfo/voiceops > _______________________________________________ > VoiceOps mailing list > VoiceOps@voiceops.org > https://puck.nether.net/mailman/listinfo/voiceops > > _______________________________________________ > VoiceOps mailing list > VoiceOps@voiceops.org > https://puck.nether.net/mailman/listinfo/voiceops > > > _______________________________________________ > VoiceOps mailing list > VoiceOps@voiceops.org > https://puck.nether.net/mailman/listinfo/voiceops -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps@voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
