Trust but verify.

We monitor and alert on:
    - Every API connection we rely on for production
    - Every SMSC connection
    - Call Metrics
    - Inbound and Outbound SMS Volume against trailing 3 weeks of data on
      the same day and time to be alerted of spikes or unexpected drops
    - Test inbound and outbound SMS regularly on every DID when there has
      not been recent activity
    - Test inbound calling to every DID when there has not been recent
      activity
    - Test outbound carrier calling endpoints to ensure they are responding
      and reachable
    - automatedly review logs for problem indicators, spikes in errors, etc
    - and other stuff!

Successes:

    - Detected a situation where a tier 0 carrier was completely unaware
      that there would be 20-40 minute drops in a bind to their inbound
      short code carrier/partner that we highlighted and detected based on
      our trailing 3 weeks of comparative data based on a small volume.

    - Alerted staff to periods of API unavailability that their own
      monitoring missed, due to a Web Application Firewall bug

    - Caught countless situations where a number ported away, or an NNID
      assignment was "lost" and our carrier did not detect nor notify us of
      these actions

I'm fascinated by the automation of call quality and packet loss in calls,
and look forward to hearing more from other people about how and what they
are monitoring, and stories about how that proactive testing has found
issues and improved customer experience.

:-)

Beckman, operationally nerdy


On Wed, 4 Nov 2026, Mike Hammett via VoiceOps wrote:

Hi all,



We just chased two weeks of intermittent inbound RTP loss (carrier toward our 
SBC, business hours only) that nothing we had could see. Our SBC's counters and 
VoIPmonitor agreed on what arrived; the carrier's egress probe showed zero 
loss, so the loss was somewhere on the public path between them, and the 
carrier called it clean. We are also mid-migration from Metaswitch to 
NetSapiens, so whatever we set up has to make sense on both.



Before this, we had VoIPmonitor on a mirror of the SBC traffic and Smokeping 
and LibreNMS watching our own routers, which told us nothing about the carrier 
path. What we are putting in now: MOS and loss alerts in VoIPmonitor, Smokeping 
with call-shaped pings (1 per second, 172 bytes, EF) to every hop toward each 
carrier that answers, and SIP OPTIONS every minute to the carriers' SBCs (they 
answer with a 500, which is fine; it is a timed reply over the real path). The 
gap is anything that measures inbound traffic to us continuously.



I am after persistent monitoring people have left running for months, not 
one-off tests. A few things I would like to hear about:

- Scheduled test calls through the carrier, scored by your passive monitor. Who 
places them, how often, and did it ever catch something before customers did?

- Carriers that give you ongoing quality data: RTCP from their side, a looking 
glass, per-trunk stats. Does anyone get this without a fight?

- Measuring one-way loss toward you. A VM near the carrier running irtt or 
similar? Something a NetSapiens shop already does out of the box?

- OPTIONS to carrier SBCs as a standing probe. Any carrier pushback?



The 2023 "troubleshooting RTP loss" thread covered the one-off side (mtr, sized 
pings, Telchemy). I will summarize the replies and add them to the list.



Thanks,
Mike







----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com 
Midwest Internet Exchange http://www.midwest-ix.com

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
[email protected]                                https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list -- [email protected]
https://lists.voiceops.org/postorius/lists/voiceops.voiceops.org/
To unsubscribe send an email to [email protected]

Reply via email to