We've recently run into a problem here that I wanted to share as a cautionary
tale.
We're in the process of upgrading our network design to be more decentralized and fault tolerant. In
planning for this we realized that converting CCA to out of band would compliment the new network
design as well as ease some architectural challenges. So we've been migrating a building at a time
to the new network model and OOB concurrently.
Recently, we started seeing two apparently separate problems. Some users would log in to CCA as per
normal, but then get bumped back to the authentication VLAN and prompted to log in again. Other
users would log in, receive errors and then be unable to receive any network traffic at all. Looking
at the switch and CCA logs we were able to determine that the first issue was caused by an
additional link-down trap being sent after successful login, in response to which CCA was resetting
the port (as it should). The second issue was being caused by ports being set to disabled, but there
was no clear reason why.
In troubleshooting this further with Cisco TAC, we found that on the few Catalyst 4000 model
switches we had migrated, utilization was extremely high. This was causing them to be unresponsive
to SNMP traffic and process SNMP commands out of order, or drop commands entirely (causing the port
bouncing and disabling we were seeing). However, we also found out that when this happens a great
deal with one or two switches, the CAM will get overwhelmed and will start dropping SNMP
communication with other switches, causing the problem to cascade through-out the network (even on
newer hardware).
The solution we have currently implemented is to switch to MAC based notification instead of link
based notification, which is configured both in the switch settings in the CAM and in the switch
configurations themselves. This should lighten the SNMP load enough that utilization comes down on
both the 4000s and the CAM. Currently it seems to be working.
The takeaways that I wanted to pass on to the list are that MAC based notification seems to be a
better option in terms of lower overhead (both network and CAM) and that the older 4000 switches,
while supported, don't work well with OOB and should be upgraded if possible before deployment.
--
Isabelle Graham
Information Security
American University
