Send Outages-discussion mailing list submissions to
outages-discussion@outages.org
To subscribe or unsubscribe via the World Wide Web, visit
https://puck.nether.net/mailman/listinfo/outages-discussion
or, via email, send a message with subject or body 'help' to
outages-discussion-requ...@outages.org
You can reach the person managing the list at
outages-discussion-ow...@outages.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Outages-discussion digest..."
Today's Topics:
1. Re: [outages] Second Bandwidth Outage Sunday (Jay Hennigan)
2. Re: [outages] Second Bandwidth Outage Sunday (Charles Sprickman)
3. Re: [outages] Second Bandwidth Outage Sunday
(Aryn Nakaoka 808.356.2901)
----------------------------------------------------------------------
Message: 1
Date: Mon, 27 Sep 2021 11:52:49 -0700
From: Jay Hennigan <j...@west.net>
To: outages-discussion@outages.org
Subject: Re: [Outages-discussion] [outages] Second Bandwidth Outage
Sunday
Message-ID: <0e4ff548-a037-b2f5-45a1-8f41ea1da...@west.net>
Content-Type: text/plain; charset=utf-8; format=flowed
On 9/27/21 11:42, Scott Harvanek wrote:
> Is there anyone with direct peering to Bandwidth that is having the same
> issues?
We don't but I'm in touch with a very high-level guy there who says that
it would not help in the current situation.
> We are in DA3 but do not have a x-connect over to Bandwidth but I may
> reconsider that if it's helping people avoid these issues...
They've productized this so you need to go through your account rep and
there may be a charge for it. They call it 'DLS,' which is you coming to
their port, or 'NES,' they come to your port.
--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV
------------------------------
Message: 2
Date: Mon, 27 Sep 2021 14:55:50 -0400
From: Charles Sprickman <sp...@bway.net>
To: Jared Geiger <ja...@compuwizz.net>
Cc: Outages Discussion <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Second Bandwidth Outage
Sunday
Message-ID: <6e3c109f-eba1-4557-b22d-14430ab03...@bway.net>
Content-Type: text/plain; charset="us-ascii"
> On Sep 27, 2021, at 2:31 PM, Jared Geiger <ja...@compuwizz.net> wrote:
>
> It looks like Bandwidth dropped all their BGP peers except NTT, presumably to
> use NTT's DDoS services.
How many do they typically have?
>
> As far as peering, Bandwidth offers connectivity via Megaport. Peerless
> (21808) is on a bunch of Equinix exchanges https://peeringdb.com/net/16256
> <https://peeringdb.com/net/16256>. Inteliquent (40160) is on some exchanges
> as well https://peeringdb.com/net/18236 <https://peeringdb.com/net/18236>.
>
> I did try and get Peerless to join Coresite Any2East last year because they
> are in the same building but they said no.
>
> On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought <douglas.vou...@themsgctr.com
> <mailto:douglas.vou...@themsgctr.com>> wrote:
> They are. Several customers have contacted me demanding that I plug the phone
> line back in.
>
> On 9/27/2021 13:54, Drew Matthews via Outages wrote:
>> i believe they are having issues again, not sure if its ddos related. but
>> they are experiencing some major issues. https://status.bandwidth.com/
>> <https://status.bandwidth.com/>
>>
>> Update - Bandwidth teams are also investigating delays with Toll-Free
>> Messaging at this time along with V2 Voice, Standard Voice, 911 traffic,
>> Portal & API with intermittent service disruptions.
>> Sep 27, 13:01 EDT
>> Update - We continue to investigate this incident.
>> Sep 27, 11:57 EDT
>> Update - Bandwidth teams are also investigating delayed ports at this time
>> along with V2 Voice, Standard Voice, 911 traffic, Portal & API with
>> intermittent service disruptions.
>> Sep 27, 11:07 EDT
>> Update - We continue to investigate this incidnet.
>> Sep 27, 11:02 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice, V2
>> Voice, 911 traffic, Portal & API with intermittent service disruption.
>> Sep 27, 10:00 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice, V2
>> Voice, and 911 traffic with intermittent service disruption.
>> Sep 27, 09:42 EDT
>> Investigating - Bandwidth teams are investigating an incident impacting our
>> V2 Voice services. Customers may notice intermittent service disruptions.
>> Sep 27, 09:12 EDT
>>
>> On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages
>> <outa...@outages.org <mailto:outa...@outages.org>> wrote:
>> Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right
>> at
>> 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident
>> at
>> Bandwidth. The impact to us was much reduced this time, as we saw some
>> flapping
>> of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough
>> to
>> fall below our thresholds.
>>
>> The API started acting up at 14:56:00 UTC, but recovered, and then has been
>> flapping between OK and degraded since then. As of 21:52 UTC, the API and
>> Portal/Dashboard are either non-responsive, slow, or degraded.
>>
>> Inbound Voice Traffic has remaind at expected historical levels throughout.
>> I have not checked on Termination, but it was working earlier today without
>> issue.
>>
>> Latest from BW https://status.bandwidth.com/ <https://status.bandwidth.com/>
>>
>> Update - Bandwidth teams have observed expected call behavior for 911
>> calls. 911 calls are processing as designed and are fully restored.
>> Teams
>> continue to work on standard voice traffic.
>>
>> Sep 26, 17:20 EDT
>>
>> I question the phrase "processing as designed" which implies it is not
>> working as the end user expects... odd.
>>
>> Beckman
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beck...@angryox.com <mailto:beck...@angryox.com>
>> https://www.angryox.com/ <https://www.angryox.com/>
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> Outages mailing list
>> outa...@outages.org <mailto:outa...@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
>> <https://puck.nether.net/mailman/listinfo/outages>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> outa...@outages.org <mailto:outa...@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
>> <https://puck.nether.net/mailman/listinfo/outages>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org <mailto:Outages-discussion@outages.org>
> https://puck.nether.net/mailman/listinfo/outages-discussion
> <https://puck.nether.net/mailman/listinfo/outages-discussion>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/3a1b8976/attachment-0001.htm>
------------------------------
Message: 3
Date: Mon, 27 Sep 2021 09:09:28 -1000
From: "Aryn Nakaoka 808.356.2901" <anaka...@trinet-hi.com>
To: Charles Sprickman <sp...@bway.net>
Cc: Jared Geiger <ja...@compuwizz.net>, Outages Discussion
<outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Second Bandwidth Outage
Sunday
Message-ID:
<caemzvjg4+0+m-mhm-jkphgizlducck3jqi9ncm+bmcmytdf...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Typically 3 :
https://bgp.he.net/AS12130#_graph4
- NTT
- L3
- Telia
[image: Screen Shot 2021-09-27 at 9.09.19 AM.png]
Aryn Nakaoka
anaka...@trinet-hi.com
Direct: 808.356.2901
518 Holokahana Lane
Honolulu, Hi 96817
AlohaTone Mobile: https://youtu.be/PdUyuf0hTYY
A Better Solution https://www.trinet-hi.com/abettersolution.pdf
<https://www.trinet-hi.com/abettersolution.pdf>
Aloha Tone PBX https://www.youtube.com/watch?v=96YWPY9wCeU
CONFIDENTIALITY NOTICE: The information contained in this email and any
attachments may be privileged, confidential and protected from disclosure.
Any disclosure, distribution or copying of this email or any attachments by
persons or entities other than the intended recipient is prohibited. If you
have received this email in error, please notify the sender immediately by
replying to the message and deleting this email and any attachments from
your system. Thank you for your cooperation.
On Mon, Sep 27, 2021 at 9:04 AM Charles Sprickman <sp...@bway.net> wrote:
>
>
> On Sep 27, 2021, at 2:31 PM, Jared Geiger <ja...@compuwizz.net> wrote:
>
> It looks like Bandwidth dropped all their BGP peers except NTT, presumably
> to use NTT's DDoS services.
>
>
> How many do they typically have?
>
>
> As far as peering, Bandwidth offers connectivity via Megaport. Peerless
> (21808) is on a bunch of Equinix exchanges https://peeringdb.com/net/16256.
> Inteliquent (40160) is on some exchanges as well
> https://peeringdb.com/net/18236.
>
> I did try and get Peerless to join Coresite Any2East last year because
> they are in the same building but they said no.
>
> On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought <
> douglas.vou...@themsgctr.com> wrote:
>
>> They are. Several customers have contacted me demanding that I plug the
>> phone line back in.
>> On 9/27/2021 13:54, Drew Matthews via Outages wrote:
>>
>> i believe they are having issues again, not sure if its ddos related. but
>> they are experiencing some major issues. https://status.bandwidth.com/
>>
>> Update - Bandwidth teams are also investigating delays with Toll-Free
>> Messaging at this time along with V2 Voice, Standard Voice, 911 traffic,
>> Portal & API with intermittent service disruptions.
>> Sep 27, 13:01 EDT
>> Update - We continue to investigate this incident.
>> Sep 27, 11:57 EDT
>> Update - Bandwidth teams are also investigating delayed ports at this
>> time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with
>> intermittent service disruptions.
>> Sep 27, 11:07 EDT
>> Update - We continue to investigate this incidnet.
>> Sep 27, 11:02 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice,
>> V2 Voice, 911 traffic, Portal & API with intermittent service disruption.
>> Sep 27, 10:00 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice,
>> V2 Voice, and 911 traffic with intermittent service disruption.
>> Sep 27, 09:42 EDT
>> Investigating - Bandwidth teams are investigating an incident impacting
>> our V2 Voice services. Customers may notice intermittent service
>> disruptions.
>> Sep 27, 09:12 EDT
>>
>> On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <
>> outa...@outages.org> wrote:
>>
>>> Again today, on the heels of the 2h 47m DDoS-related outage yesterday,
>>> right at
>>> 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another
>>> incident at
>>> Bandwidth. The impact to us was much reduced this time, as we saw some
>>> flapping
>>> of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive
>>> enough to
>>> fall below our thresholds.
>>>
>>> The API started acting up at 14:56:00 UTC, but recovered, and then has
>>> been
>>> flapping between OK and degraded since then. As of 21:52 UTC, the API and
>>> Portal/Dashboard are either non-responsive, slow, or degraded.
>>>
>>> Inbound Voice Traffic has remaind at expected historical levels
>>> throughout.
>>> I have not checked on Termination, but it was working earlier today
>>> without
>>> issue.
>>>
>>> Latest from BW https://status.bandwidth.com/
>>>
>>> Update - Bandwidth teams have observed expected call behavior for
>>> 911
>>> calls. 911 calls are processing as designed and are fully restored.
>>> Teams
>>> continue to work on standard voice traffic.
>>>
>>> Sep 26, 17:20 EDT
>>>
>>> I question the phrase "processing as designed" which implies it is not
>>> working as the end user expects... odd.
>>>
>>> Beckman
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beck...@angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>> _______________________________________________
>>> Outages mailing list
>>> outa...@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
>> _______________________________________________
>> Outages mailing
>> listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages-discussion mailing list
>> Outages-discussion@outages.org
>> https://puck.nether.net/mailman/listinfo/outages-discussion
>>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/fc47e620/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2021-09-27 at 9.09.19 AM.png
Type: image/png
Size: 26034 bytes
Desc: not available
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/fc47e620/attachment.png>
------------------------------
Subject: Digest Footer
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
------------------------------
End of Outages-discussion Digest, Vol 139, Issue 10
***************************************************
