Send Outages-discussion mailing list submissions to
outages-discussion@outages.org
To subscribe or unsubscribe via the World Wide Web, visit
https://puck.nether.net/mailman/listinfo/outages-discussion
or, via email, send a message with subject or body 'help' to
outages-discussion-requ...@outages.org
You can reach the person managing the list at
outages-discussion-ow...@outages.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Outages-discussion digest..."
Today's Topics:
1. Re: [outages] Second Bandwidth Outage Sunday (Jared Geiger)
2. Re: [outages] Second Bandwidth Outage Sunday (Scott Harvanek)
----------------------------------------------------------------------
Message: 1
Date: Mon, 27 Sep 2021 11:31:41 -0700
From: Jared Geiger <ja...@compuwizz.net>
To: Outages Discussion <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Second Bandwidth Outage
Sunday
Message-ID:
<cahuchrbgyhvaciptewzf-dms27-pdgmgts_ksj66ag5s_lv...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
It looks like Bandwidth dropped all their BGP peers except NTT, presumably
to use NTT's DDoS services.
As far as peering, Bandwidth offers connectivity via Megaport. Peerless
(21808) is on a bunch of Equinix exchanges https://peeringdb.com/net/16256.
Inteliquent (40160) is on some exchanges as well
https://peeringdb.com/net/18236.
I did try and get Peerless to join Coresite Any2East last year because they
are in the same building but they said no.
On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought <
douglas.vou...@themsgctr.com> wrote:
> They are. Several customers have contacted me demanding that I plug the
> phone line back in.
> On 9/27/2021 13:54, Drew Matthews via Outages wrote:
>
> i believe they are having issues again, not sure if its ddos related. but
> they are experiencing some major issues. https://status.bandwidth.com/
>
> Update - Bandwidth teams are also investigating delays with Toll-Free
> Messaging at this time along with V2 Voice, Standard Voice, 911 traffic,
> Portal & API with intermittent service disruptions.
> Sep 27, 13:01 EDT
> Update - We continue to investigate this incident.
> Sep 27, 11:57 EDT
> Update - Bandwidth teams are also investigating delayed ports at this
> time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with
> intermittent service disruptions.
> Sep 27, 11:07 EDT
> Update - We continue to investigate this incidnet.
> Sep 27, 11:02 EDT
> Update - Bandwidth teams are investigating an incident impacting Voice,
> V2 Voice, 911 traffic, Portal & API with intermittent service disruption.
> Sep 27, 10:00 EDT
> Update - Bandwidth teams are investigating an incident impacting Voice,
> V2 Voice, and 911 traffic with intermittent service disruption.
> Sep 27, 09:42 EDT
> Investigating - Bandwidth teams are investigating an incident impacting
> our V2 Voice services. Customers may notice intermittent service
> disruptions.
> Sep 27, 09:12 EDT
>
> On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <
> outa...@outages.org> wrote:
>
>> Again today, on the heels of the 2h 47m DDoS-related outage yesterday,
>> right at
>> 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another
>> incident at
>> Bandwidth. The impact to us was much reduced this time, as we saw some
>> flapping
>> of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive
>> enough to
>> fall below our thresholds.
>>
>> The API started acting up at 14:56:00 UTC, but recovered, and then has
>> been
>> flapping between OK and degraded since then. As of 21:52 UTC, the API and
>> Portal/Dashboard are either non-responsive, slow, or degraded.
>>
>> Inbound Voice Traffic has remaind at expected historical levels
>> throughout.
>> I have not checked on Termination, but it was working earlier today
>> without
>> issue.
>>
>> Latest from BW https://status.bandwidth.com/
>>
>> Update - Bandwidth teams have observed expected call behavior for 911
>> calls. 911 calls are processing as designed and are fully restored.
>> Teams
>> continue to work on standard voice traffic.
>>
>> Sep 26, 17:20 EDT
>>
>> I question the phrase "processing as designed" which implies it is not
>> working as the end user expects... odd.
>>
>> Beckman
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet
>> Guy
>> beck...@angryox.com
>> https://www.angryox.com/
>>
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> Outages mailing list
>> outa...@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> _______________________________________________
> Outages mailing
> listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/d3efce95/attachment-0001.htm>
------------------------------
Message: 2
Date: Mon, 27 Sep 2021 13:42:59 -0500
From: Scott Harvanek <scott.harva...@login.com>
To: Jared Geiger <ja...@compuwizz.net>
Cc: Outages Discussion <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Second Bandwidth Outage
Sunday
Message-ID: <ae398022-2ab4-992f-7f43-f412ebc7c...@login.com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Is there anyone with direct peering to Bandwidth that is having the same
issues?
We are in DA3 but do not have a x-connect over to Bandwidth but I may
reconsider that if it's helping people avoid these issues...
--
*
**
*
Jared Geiger wrote on 9/27/21 1:31 PM:
> It looks like Bandwidth dropped all their BGP peers except NTT,
> presumably to use NTT's DDoS services.
>
> As far as peering, Bandwidth offers connectivity via Megaport.
> Peerless (21808) is on a bunch of Equinix exchanges
> https://peeringdb.com/net/16256. Inteliquent (40160) is on some
> exchanges as well https://peeringdb.com/net/18236.
>
> I did try and get Peerless to join Coresite Any2East last year because
> they are in the same building but they said no.
>
> On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought
> <douglas.vou...@themsgctr.com <mailto:douglas.vou...@themsgctr.com>>
> wrote:
>
> They are. Several customers have contacted me demanding that I
> plug the phone line back in.
>
> On 9/27/2021 13:54, Drew Matthews via Outages wrote:
>> i believe they are having issues again, not sure if its ddos
>> related. but they are experiencing some major issues.
>> https://status.bandwidth.com/
>>
>> Update?- Bandwidth teams are also investigating delays with
>> Toll-Free Messaging at this time along with V2 Voice, Standard
>> Voice, 911 traffic, Portal & API with intermittent service
>> disruptions.
>> Sep 27, 13:01?EDT
>> Update?- We continue to investigate this incident.
>> Sep 27, 11:57?EDT
>> Update?- Bandwidth teams are also investigating delayed ports at
>> this time along with V2 Voice, Standard Voice, 911 traffic,
>> Portal & API with intermittent service disruptions.
>> Sep 27, 11:07?EDT
>> Update?- We continue to investigate this incidnet.
>> Sep 27, 11:02?EDT
>> Update?- Bandwidth teams are investigating an incident impacting
>> Voice, V2 Voice, 911 traffic, Portal & API with intermittent
>> service disruption.
>> Sep 27, 10:00?EDT
>> Update?- Bandwidth teams are investigating an incident impacting
>> Voice, V2 Voice, and 911 traffic with intermittent service
>> disruption.
>> Sep 27, 09:42?EDT
>> Investigating?- Bandwidth teams are investigating an incident
>> impacting our V2 Voice services. Customers may notice
>> intermittent service disruptions.
>> Sep 27, 09:12?EDT
>>
>> On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages
>> <outa...@outages.org <mailto:outa...@outages.org>> wrote:
>>
>> Again today, on the heels of the 2h 47m DDoS-related outage
>> yesterday, right at
>> 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to
>> another incident at
>> Bandwidth. The impact to us was much reduced this time, as we
>> saw some flapping
>> of their SIP Inbound SBCs, but around 15:20:12 UTC it was
>> responsive enough to
>> fall below our thresholds.
>>
>> The API started acting up at 14:56:00 UTC, but recovered, and
>> then has been
>> flapping between OK and degraded since then. As of 21:52 UTC,
>> the API and
>> Portal/Dashboard are either non-responsive, slow, or degraded.
>>
>> Inbound Voice Traffic has remaind at expected historical
>> levels throughout.
>> I have not checked on Termination, but it was working earlier
>> today without
>> issue.
>>
>> Latest from BW https://status.bandwidth.com/
>>
>> ? ? ?Update - Bandwidth teams have observed expected call
>> behavior for 911
>> ? ? ?calls. 911 calls are processing as designed and are
>> fully restored. Teams
>> ? ? ?continue to work on standard voice traffic.
>>
>> ? ? ?Sep 26, 17:20 EDT
>>
>> I question the phrase "processing as designed" which implies
>> it is not
>> working as the end user expects... odd.
>>
>> Beckman
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beck...@angryox.com <mailto:beck...@angryox.com>
>> https://www.angryox.com/
>>
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> Outages mailing list
>> outa...@outages.org <mailto:outa...@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> _______________________________________________
>> Outages mailing list
>> outa...@outages.org <mailto:outa...@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org <mailto:Outages-discussion@outages.org>
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
>
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/ec021b31/attachment.htm>
------------------------------
Subject: Digest Footer
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
------------------------------
End of Outages-discussion Digest, Vol 139, Issue 9
**************************************************
