Hmmm... DoS? -----Original Message----- From: VoiceOps <voiceops-boun...@voiceops.org> On Behalf Of Peter Beckman via VoiceOps Sent: Saturday, September 25, 2021 4:49 PM To: VoiceOps <voiceops@voiceops.org> Subject: Re: [VoiceOps] Bandwidth East Coast Issues
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs 67.231.9.112 67.231.13.113 On Sat, 25 Sep 2021, Peter Beckman wrote: > Another Update > > Update - Bandwidth teams continue to investigate Voice and Messaging > Failures. 911 calls are experiencing failures. > > Customers may also experience portal and portal API time-outs, > slowness, or > unresponsiveness. > > All teams are working towards a resolution at this time. > > Sep 25, 16:34 EDT > > Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm > EDT) as they are now responding at an acceptable rate. I just tested > outbound and got a call through to a Verizon Mobile, though the PDD > was about 5-6 seconds. > > Inbound seems to be working, albeit a bit slower than expected. > > Seems like we're on the path to full recovery. > > Beckman > > On Sat, 25 Sep 2021, Peter Beckman wrote: > >> After a long silence, no updates other than "we're working on it." >> >> Update - Bandwidth teams continue to investigate Voice and Messaging >> Failures. >> >> At this time, 911 calls are not impacted. >> >> Customers may also experience portal and portal API time-outs, >> slowness, or unresponsiveness. >> >> All teams are working towards a resolution at this time. >> >> Sep 25, 16:22 EDT >> >> We are still seeing some calls get through. SMS traffic is being >> delivered faster and we have not seen any missed SMS messages. Worst >> SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all >> under 21 seconds, 3 of 5 were under 2 seconds. >> >> About half of our calls getting through are Canada DIDs. >> >> Beckman >> >> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote: >> >>> As of 3:31pm EDT Bandwidth has acknowledged the issue: >>> >>> Investigating - Bandwidth is investigating an incident impacting Voice >>> and Messaging Services. Calls and Messages may experience unexpected >>> failures. All teams are actively engaged. >>> >>> Sep 25, 15:31 EDT >>> >>> I will update here as I get updates. >>> >>> Outbound calls still failing with timeouts, Inbound calls failing, >>> SMS messages are delayed, the delay now up to 8 minutes. >>> >>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote: >>> >>>> We began detecting issues with Bandwidth Inbound and Outbound Voice >>>> and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep >>>> has acknowledged issues are detected but their status page as of >>>> yet has not been updated. >>>> >>>> I'm seeing outbound calls time out, inbound calls to BW DIDs >>>> receive a "call cannot be completed" error on Verizon, and our SMS >>>> tests every 10 minutes that usually complete in under 2 seconds are >>>> taking 15 seconds to >>>> 3.65 minutes to reach us. >>>> >>>> Hope they find the culprit and get it fixed quickly! >>>> >>>> As of 3:27:45pm EDT their status page shows zero incidents >>>> >>>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidt >>>> h.com%2f&c=E,1,0374XKPpekgcDcgZahsM1KMmqs8G8yvMF9tnfZpRJpXXTxrgL2-V >>>> igndLyRQtrrMJGWJ6i1KaXs_XiCvMeXdhv1lhddgpQqt4YCMx53FElYAtdl7Uq2rzdD >>>> ogA,,&typo=1 >>> _______________________________________________ >>> VoiceOps mailing list >>> VoiceOps@voiceops.org >>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net% >>> 2fmailman%2flistinfo%2fvoiceops&c=E,1,d6hh1sPNldOI_pqt_AGZFGNoKoWaZU >>> bc84nufxrulb109riOKTWRby6yDeUYeLJQ0GbVnCNv0oy1_Tzugdlt5gWrQyeF7siUkS >>> yZdGoiE5g,&typo=1 >>> >> >> --------------------------------------------------------------------------- >> Peter Beckman Internet Guy >> beck...@angryox.com >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1,rbn4pLGIqtePoufgSG0gpI7tx3qNUxYkzxKybtyGo25HFwBHxTfqzDSJg1H_qmBIXOjOH_b4tu1C5cz96CBpsy7q6D1eiz4LWuisYlckAEDvry-sBDZubw,,&typo=1 >> --------------------------------------------------------------------- >> ------ >> > > --------------------------------------------------------------------------- > Peter Beckman Internet Guy > beck...@angryox.com > https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1,h-PZ56YcwFzh2h6U3OBrBfL5rynQyYW5r0MlA8zMv4sNnAfqCU4pZ_9YgrauFWYKZLrjaKoPNUsWlQXj28km8xOd0dN7RvxhrdCBocSWFsdIUFHMOEjHWA,,&typo=1 > ---------------------------------------------------------------------- > ----- > --------------------------------------------------------------------------- Peter Beckman Internet Guy beck...@angryox.com https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1,lT8OseOtcOeZQiLP2hkAYErLoNRmyvW7M4KFJ56BvNCSlN7ikWFKZ4H7cNZ2c64nu08oBE-eFm4iaKsZItbUYlobkcJiVcEaTWCDGwUVNA,,&typo=1 --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps@voiceops.org https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2fmailman%2flistinfo%2fvoiceops&c=E,1,8Cb0Xq6iM0kFLdX8GNjgCSSWWSI33AVhXAoOUeNqDwe6J7iEp6leFZ_rnR_bjvjme2lamRMwBbGXZUVmkvFI--HcGQNNDcKk_1ucUnaWm259fMr6OMkIil9K&typo=1 _______________________________________________ VoiceOps mailing list VoiceOps@voiceops.org https://puck.nether.net/mailman/listinfo/voiceops