It ended up being the network that was causing the issue, we provided numerous packet sniffer logs showing that the snmp packets were being sent but the devices were not responding to them, the packets were lost in transmission(the main cause), or in some cases responding with malformed SNMP packets. We were having a lot of issues and the main trouble spot was a load balancer that ran out of nat ports from time to time. Our traffic is no longer "natted" with pat and this issue went away for us. It still comes up from time to time, but now that we can explain what Spectrum is doing, there aren't many questions or fingers pointing at spectrum for causing the management agent lost alarms.
Spectrum seems from my limited testing to send an snmp packet requesting the sysuptime/hrSystemUptime it can do others also it seems for the poll check, during the devices polling period dependent on polling group etc in Spectrum. If we don't receive a response or get an ICMP port unreachable or other error conditions, spectrum will try again at the DCM retry rate. Once the retry failure rate is exceeded it sends an icmp echo request, if that fails it keeps trying at a defined rate and will pop a device has stopped responding to polls/pings instead of the management agent lost. If the ICMP pings succeed then it pops the management agent lost. Many things can affect this, and I did my testing on 8.1.x verifying it only for a few minutes in 9.1.x so things may change and the above discussion is for a device modeled in spectrum as a systemEdge device model. I would still recommend that you run a sniffer, each specific device could have a different issue or the same issue or my explanation could be way off. We have once seen a spectroserver cause all of the devices modeled with snmp access on it to have management agent lost alarms, and the network did not have an issue at the time, we had to restart the spectroserver and the issue went away. It was only once though, and on the 8.1.x version. Hopefully I provided some useful information and if you have any questions then please let me know. Regards, Sean Sean Carnes Cablevision Systems Corp. Network Management Systems & Metrics 172*85*77/5167902323/5163905910 >>> "Hofmann, Berthold" <[email protected]> 2/24/2011 9:36 AM >>> Hi Sean, can you please explain a little bit what you have found, and did you find a solution to resolve this. Because we have the same problem. For my opinion it seems that older devices have the problem more then newer devices. I think it is a problem from the cpu resource of older devices. Best Regards Berthold -----Ursprüngliche Nachricht----- Von: Sean Carnes [mailto:[email protected]] Gesendet: Donnerstag, 24. Februar 2011 15:08 An: spectrum Betreff: Re: [spectrum] Sensitivity to agent lost events Kenneth, We had some issues with explaining exactly what Spectrum was doing to determine that the device stopped responding to polls. The approach we took to determine the behavior, other than what is in the manual, was to run a packet sniffer and filter for traffic to/from the device/devices in question. It was quite educational and a little bit different than we expected. Regards, Sean Sean Carnes Cablevision Systems Corp. Network Management Systems & Metrics 172*85*77/5167902323/5163905910 >>> Kenneth Kirchner <[email protected]> 2/24/2011 1:43 AM >>> Does anyone have any advice on how to make Spectrum less sensitive to "Management Agent Lost" events? We see lots of events where Spectrum says the agent is lost, but comes up less than 3 minutes later (poll interval is 300 seconds, so I am not sure how it even knows). If we manually poll as soon as we get the alarm, 99% of the time it is successful. I have raised the DCM Retry Count to 3, but that does not appear to have helped. We have all kinds of WAN links (ATM, T1, etc) and the latency/reliability issues that go with them, so I cant say the network is not at fault, but I wish I had more information about what Spectrum tried and when it tried it. As soon as we get the Major alarm, we get a ticket in our ticketing system, so this is creating a bit of extra work for a problem that has resolved itself before the ticket even gets assigned. -K --- To unsubscribe from spectrum, send email to [email protected] with the body: unsubscribe spectrum [email protected] *--To unsubscribe from spectrum, send email to [email protected] with the body: unsubscribe spectrum [email protected] -------------------------------------------------------- The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain information concerning Cablevision and/or its affiliates and subsidiaries that is proprietary, privileged, confidential and/or subject to copyright. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof. -------------------------------------------------------- -------------------------------------------------------- The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain information concerning Cablevision and/or its affiliates and subsidiaries that is proprietary, privileged, confidential and/or subject to copyright. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof. -------------------------------------------------------- --- To unsubscribe from spectrum, send email to [email protected] with the body: unsubscribe spectrum [email protected]
