After having some problems switching to 64-bit counters (in a different thread), I realized it was a symptom rather than the problem itself. So I rolled back to a previous backup before any changes were made and started looking for a root cause.
Basically, I went back to the configuration before I started SNMP monitoring my core routers. They exist as devices in Zenoss but SNMP monitoring is off. At this point, everything looks normal, and zenperfsnmp runs flawlessly on the other devices (a number of hosts and a few switches). Once I switch on SNMP monitoring of the (13) core routers is when the problems start. It appears from the Zenoss web-interface that everything is in order (which is why I never noticed there was a problem initially), but when I run zenperfsnmp from the command line, I get results similar to this: Code: [EMAIL PROTECTED] ~]$ zenperfsnmp run WARNING:zen.zenperfsnmp:Reconnected to ZenHub INFO:zen.zenperfsnmp:fetching property items INFO:zen.zenperfsnmp:getting threshold classes INFO:zen.zenperfsnmp:checking for outdated configs INFO:zen.zenperfsnmp:fetching configs for []... INFO:zen.zenperfsnmp:fetching default RRDCreateCommand INFO:zen.zenperfsnmp:getting collector thresholds INFO:zen.zenperfsnmp:fetching snmp status INFO:zen.zenperfsnmp:Initiating incremental device load INFO:zen.zenperfsnmp:total=26 good=0 bad=0 time=0.000061 INFO:zen.zenperfsnmp:total=26 good=1 bad=0 time=0.243458 INFO:zen.zenperfsnmp:total=26 good=2 bad=0 time=0.251428 INFO:zen.zenperfsnmp:total=26 good=3 bad=0 time=0.265071 INFO:zen.zenperfsnmp:total=26 good=4 bad=0 time=0.328599 INFO:zen.zenperfsnmp:total=26 good=5 bad=0 time=0.412457 INFO:zen.zenperfsnmp:total=26 good=6 bad=0 time=0.430121 INFO:zen.zenperfsnmp:total=26 good=7 bad=0 time=0.436873 INFO:zen.zenperfsnmp:total=26 good=8 bad=0 time=0.463705 INFO:zen.zenperfsnmp:total=26 good=9 bad=0 time=0.473675 INFO:zen.zenperfsnmp:total=26 good=10 bad=0 time=0.525798 INFO:zen.zenperfsnmp:total=26 good=11 bad=0 time=0.550150 INFO:zen.zenperfsnmp:total=26 good=12 bad=0 time=0.570894 INFO:zen.zenperfsnmp:total=26 good=13 bad=0 time=0.592087 INFO:zen.zenperfsnmp:total=26 good=14 bad=0 time=0.598951 INFO:zen.zenperfsnmp:total=26 good=15 bad=0 time=0.621365 INFO:zen.zenperfsnmp:total=26 good=16 bad=0 time=0.662331 INFO:zen.zenperfsnmp:total=26 good=17 bad=0 time=0.694719 INFO:zen.zenperfsnmp:total=26 good=18 bad=0 time=0.731580 INFO:zen.zenperfsnmp:total=26 good=19 bad=0 time=0.769454 INFO:zen.zenperfsnmp:total=26 good=20 bad=0 time=0.833187 INFO:zen.zenperfsnmp:total=26 good=21 bad=0 time=0.890530 INFO:zen.zenperfsnmp:total=26 good=22 bad=0 time=0.944207 INFO:zen.zenperfsnmp:total=26 good=23 bad=0 time=1.025372 INFO:zen.zenperfsnmp:total=26 good=24 bad=0 time=1.054623 INFO:zen.zenperfsnmp:total=26 good=25 bad=0 time=1.216546 INFO:zen.zenperfsnmp:sent 2786 OID requests INFO:zen.zenperfsnmp:collected 26 of 26 devices in 2.59 INFO:zen.zenperfsnmp:total=26 good=26 bad=0 time=2.586185 INFO:zen.zenperfsnmp:Daemon zenperfsnmp shutting down WARNING:zen.zenperfsnmp:Error loading config for devices ['cd-ti01-cor01', 'cd-sk01-cor01'] ERROR:zen.zenperfsnmp:Error loading devices: [Failure instance: Traceback (failure with no frames): twisted.spread.pb.DeadReferenceError: Calling Stale Broker ] INFO:zen.zenperfsnmp:zenperfsnmp shutting down Important things to note: - There is always two devices in the "Error loading devices" field. But they are rarely the same. It appears completely random what devices fail. - In fact, it does not appear that the devices actually fail at all, at least not completely, because data is still collected for them. - Some times, devices will show up in the "Error loading devices" field that are not the core routers, devices that worked perfectly before --- and as far as I can tell still do. I'm honestly at a loss as to what is causing this problem. And equally at a loss as to how to fix it. I really hope for some advice on how to proceed with trouble shooting or working around the issue. -------------------- m2f -------------------- Read this topic online here: http://forums.zenoss.com/viewtopic.php?p=26951#26951 -------------------- m2f -------------------- _______________________________________________ zenoss-users mailing list [email protected] http://lists.zenoss.org/mailman/listinfo/zenoss-users
