BW Considers the incident resolved. Permalink to Status Page for the outage:
https://status.bandwidth.com/incidents/d09cdj4qljc1
Final Update:
Resolved - With continued network stability, we consider this incident
resolved.
Sep 25, 2021 - 19:45 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
BW Status Page is showing 100% operational now. Our monitoring confirms
their supposition.
Monitoring - Bandwidth teams have seen overall network stability with
expected traffic behavior. We will continue to monitor the network.
Sep 25, 19:29 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Another Update
Update - Bandwidth continues to see portal and network improvements. At
this time, 911 & messaging traffic is processing as expected. We
continue to work to resolve other impacted services.
Sep 25, 19:07 EDT
The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound
and Outbound is still entirely down. Our tests show that Inbound and
Outbound are functioning, as is SMS messaging and the API.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
According to BW TAC it was a DDoS.
The outages list posted this:
Bandwidth is currently experiencing a DDoS attack which is
intermittently impacting our network. Our network operations and
engineering teams are actively engaged in critical efforts to protect
our network and fully restore all services as rapidly as possible.
On Sat, 25 Sep 2021, Peter Beckman wrote:
Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
19:00 to 21:47 UTC
We haven't completely determined whether or not this was a full or
partial
outage, as we still saw some calls come in, but termination had issues as
did SMS delivery for a portion of the outage period.
Latest BW Update
Update - Bandwidth continues to see portal and network improvements.
At
this time, messaging traffic is processing as expected. We continue to
work to resolve other impacted services.
Sep 25, 18:42 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Looks like things are recovering.
Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30
minutes
ago.
Latest Update
Identified - Bandwidth has identified the cause of this incident and
is
observing improved network and portal behavior.
Sep 25, 18:18 EDT
All of my alarms have cleared, SMS messages are coming in reliably,
Inbound
and Outbound calling seem restored in our tests.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38
UTC.
Monitoring these two IPs
67.231.9.112
67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging
Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs,
slowness, or
unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm
EDT)
as they are now responding at an acceptable rate. I just tested
outbound
and got a call through to a Verizon Mobile, though the PDD was about
5-6
seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and
Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API
time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being
delivered
faster and we have not seen any missed SMS messages. Worst SMS delay
was 9m
37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3
of 5
were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting
Voice
and Messaging Services. Calls and Messages may experience
unexpected
failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing,
SMS
messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice
and SMS
starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
acknowledged
issues are detected but their status page as of yet has not been
updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs
receive a
"call cannot be completed" error on Verizon, and our SMS tests
every 10
minutes that usually complete in under 2 seconds are taking 15
seconds to
3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
https://status.bandwidth.com/
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Peter Beckman
Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Peter Beckman Internet
Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Peter Beckman Internet
Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
---------------------------------------------------------------------------
Peter Beckman Internet
Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Peter Beckman Internet
Guy
beck...@angryox.com
https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
