Hello, CPU started to increase again with my clients' load. It's incredible how CPU variations and interface output drops (overruns) match.
Here is the output : LNS1.IX1#sh processes cpu sorted CPU utilization for five seconds: 55%/51%; one minute: 58%; five minutes: 59% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 167 148804928 670139211 222 1.19% 0.93% 0.79% 0 LFDp Input Proc 92 84592280 658514047 128 0.55% 0.47% 0.40% 0 IP Input 158 29971432 19506363 1536 0.47% 0.51% 0.51% 0 CEF: IPv4 proces 163 1816683172109814 0 0.23% 0.17% 0.16% 0 HQF Input Shaper 162 2110923172104782 0 0.23% 0.33% 0.33% 0 HQF Shaper Backg 237 3969572 303945705 13 0.15% 0.15% 0.15% 0 L2X Data Daemon 230 13763816 2561996 5372 0.15% 0.19% 0.18% 0 Compute load avg 44 21924884 2508789 8739 0.15% 0.15% 0.15% 0 Net Background 2 99800 2559364 38 0.07% 0.02% 0.00% 0 Load Meter 84 61420 399291246 0 0.07% 0.03% 0.02% 0 ACCT Periodic Pr 260 1430888 404901464 3 0.07% 0.09% 0.08% 0 PPP Events 238 4055340 17930027 226 0.07% 0.05% 0.06% 0 L2TP mgmt daemon 13 904 28924 31 0.00% 0.00% 0.00% 0 DDR Timers 14 60 28 2142 0.00% 0.00% 0.00% 0 Entity MIB API 12 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager 16 236 4 59000 0.00% 0.00% 0.00% 0 PrstVbl 17 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers 11 0 126 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT 15 24 346 69 0.00% 0.00% 0.00% 0 EEM ED Syslog 18 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa 21 92 213290 0 0.00% 0.00% 0.00% 0 IPC Dynamic Cach 22 58000 2556710 22 0.00% 0.00% 0.00% 0 IPC Service NonC 23 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager 24 30880 12782729 2 0.00% 0.00% 0.00% 0 IPC Periodic Tim 19 0 2 0 0.00% 0.00% 0.00% 0 ATM AutoVC Perio 26 29152 12782728 2 0.00% 0.00% 0.00% 0 IPC Deferred Por 20 0 2 0 0.00% 0.00% 0.00% 0 ATM VC Auto Crea 28 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Control 25 0 1 0 0.00% 0.00% 0.00% 0 IPC Managed Time 30 132 1279724 0 0.00% 0.00% 0.00% 0 Compute SRP rate 31 4255676 12785527 332 0.00% 0.00% 0.00% 0 EnvMon 32 8 3 2666 0.00% 0.00% 0.00% 0 OIR Handler 33 0 1 0 0.00% 0.00% 0.00% 0 Crash writer 34 0 1 0 0.00% 0.00% 0.00% 0 Exception contro 35 10708 2558103 4 0.00% 0.00% 0.00% 0 ALARM_TRIGGER_SC 36 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API 37 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun 38 35512 12782719 2 0.00% 0.00% 0.00% 0 GraphIt 39 0 2 0 0.00% 0.00% 0.00% 0 Dialer event 27 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager 41 0 1 0 0.00% 0.00% 0.00% 0 RFS server proce 42 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect 43 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd 29 5344 2559365 2 0.00% 0.00% 0.00% 0 IPC Loadometer 45 0 2 0 0.00% 0.00% 0.00% 0 IDB Work 46 8 553 14 0.00% 0.00% 0.00% 0 Logger 47 50724 12782714 3 0.00% 0.00% 0.00% 0 TTY Background 40 0 2 0 0.00% 0.00% 0.00% 0 SMART 49 3987404 220452 18087 0.00% 0.01% 0.00% 0 Per-minute Jobs 50 7512 43999 170 0.00% 0.00% 0.00% 0 IF-MGR control p 51 0 37 0 0.00% 0.00% 0.00% 0 IF-MGR event pro 52 12 3 4000 0.00% 0.00% 0.00% 0 Inode Table Dest 53 124 386718 0 0.00% 0.00% 0.00% 0 Transport Port A 54 0 2 0 0.00% 0.00% 0.00% 0 INTR MGR PROCESS 55 603356 3723660 162 0.00% 0.00% 0.00% 0 HC Counter Timer PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 56 0 2 0 0.00% 0.00% 0.00% 0 CES Timer 57 0 1 0 0.00% 0.00% 0.00% 0 POS APS Event Pr 58 0 1 0 0.00% 0.00% 0.00% 0 SONET alarm time 59 0 2 0 0.00% 0.00% 0.00% 0 Hawkeye Backgrou 60 0 1 0 0.00% 0.00% 0.00% 0 CSP Timer 61 0 1 0 0.00% 0.00% 0.00% 0 FPD Management P 62 0 1 0 0.00% 0.00% 0.00% 0 FPD Action Proce 63 0 2 0 0.00% 0.00% 0.00% 0 VNM DSPRM MAIN 64 0 1 0 0.00% 0.00% 0.00% 0 CES Line Conditi 65 0 9 0 0.00% 0.00% 0.00% 0 Flash MIB Update 66 20 1283 15 0.00% 0.00% 0.00% 0 Flash Card Oir 67 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 68 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 69 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 10 0 1 0 0.00% 0.00% 0.00% 0 ATM ASYNC PROC 71 228 213290 1 0.00% 0.00% 0.00% 0 DFS flush period 72 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 73 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 74 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 75 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 76 104 81025 1 0.00% 0.00% 0.00% 0 DFS flush period 48 140184 12782976 10 0.00% 0.00% 0.00% 0 Per-Second Jobs 79 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 80 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 81 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 9 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer 83 25516 57044 447 0.00% 0.00% 0.00% 0 AAA ACCT Proc 8 4572 13343910 0 0.00% 0.00% 0.00% 0 ARP Background 85 0 2 0 0.00% 0.00% 0.00% 0 AAA System Acct 86 4660 32163 144 0.00% 0.00% 0.00% 0 Collection proce 70 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 88 0 2 0 0.00% 0.00% 0.00% 0 CEF switching ba 89 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input 90 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER 91 12 15 800 0.00% 0.00% 0.00% 0 IP ARP Adjacency 7 1140 3275 348 0.00% 0.00% 0.00% 0 ARP Input 93 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl 94 0 4 0 0.00% 0.00% 0.00% 0 TurboACL 95 0 2 0 0.00% 0.00% 0.00% 0 TurboACL chunk 96 0 2 0 0.00% 0.00% 0.00% 0 ADJ resolve proc 97 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Echo event 98 12 21298 0 0.00% 0.00% 0.00% 0 MOP Protocols 99 0 3 0 0.00% 0.00% 0.00% 0 PPP SIP 100 81496 326157 249 0.00% 0.00% 0.00% 0 PPP Bind 101 117984 57625 2047 0.00% 0.02% 0.00% 0 PPP IP Route 102 0 1 0 0.00% 0.00% 0.00% 0 L2X Socket proce 103 225400 2360322 95 0.00% 0.00% 0.00% 0 SSM connection m 104 942996 1387675 679 0.00% 0.01% 0.00% 0 SSS Manager 105 0 1 0 0.00% 0.00% 0.00% 0 SSS Policy Manag 106 10232 115343 88 0.00% 0.00% 0.00% 0 SSS Feature Mana 107 10940 49989631 0 0.00% 0.00% 0.00% 0 SSS Feature Time 108 0 2 0 0.00% 0.00% 0.00% 0 Spanning Tree 109 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA 110 0 1 0 0.00% 0.00% 0.00% 0 delayed evt hand 111 0 1 0 0.00% 0.00% 0.00% 0 AC Switch 112 0 1 0 0.00% 0.00% 0.00% 0 SG CMD HANDLER 113 0 1 0 0.00% 0.00% 0.00% 0 IMA PROC PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 114 0 2 0 0.00% 0.00% 0.00% 0 IP PORTBUNDLE 115 0 2 0 0.00% 0.00% 0.00% 0 IPAM/ODAP Events 6 0 2 0 0.00% 0.00% 0.00% 0 Timers 117 0 2 0 0.00% 0.00% 0.00% 0 IPAM Events 118 0 1 0 0.00% 0.00% 0.00% 0 LSP Tunnel FRR 5 5824 11228 518 0.00% 0.00% 0.00% 0 Pool Manager 4 57020588 3390149 16819 0.00% 0.56% 0.48% 0 Check heaps 121 0 1 0 0.00% 0.00% 0.00% 0 AAA EPD HANDLER 122 0 1 0 0.00% 0.00% 0.00% 0 PM EPD API 123 61856 397922740 0 0.00% 0.04% 0.02% 0 RADIUS Proxy 124 0 1 0 0.00% 0.00% 0.00% 0 SG DPM 125 0 1 0 0.00% 0.00% 0.00% 0 Timer handler fo 126 0 1 0 0.00% 0.00% 0.00% 0 Prepaid response 127 0 1 0 0.00% 0.00% 0.00% 0 Timed Policy act 128 0 1 0 0.00% 0.00% 0.00% 0 AAA response han 129 0 1 0 0.00% 0.00% 0.00% 0 TC SIP 130 2816 256320 10 0.00% 0.00% 0.00% 0 CEF background p 77 0 1 0 0.00% 0.00% 0.00% 0 Async write proc 132 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute 133 0 2 0 0.00% 0.00% 0.00% 0 ATIP_UDP_TSK 134 0 3 0 0.00% 0.00% 0.00% 0 SNMP Timers 135 172872 404340 427 0.00% 0.01% 0.00% 0 IP Background 136 177848 208632 852 0.00% 0.00% 0.00% 0 IP RIB Update 137 0 2 0 0.00% 0.00% 0.00% 0 PPP Compress Inp 138 0 2 0 0.00% 0.00% 0.00% 0 PPP Compress Res 139 112392 2000493 56 0.00% 0.00% 0.00% 0 CDP Protocol 140 73824 6159 11986 0.00% 0.00% 0.00% 0 AAA Per-User 141 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers 142 3616 1689493 2 0.00% 0.00% 0.00% 0 TCP Timer 143 108024 191591 563 0.00% 0.00% 0.00% 0 TCP Protocols 144 0 1 0 0.00% 0.00% 0.00% 0 COPS 87 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R 146 14412 163001 88 0.00% 0.00% 0.00% 0 static 147 0 3 0 0.00% 0.00% 0.00% 0 Flow Exporter Ti 148 8 42659 0 0.00% 0.00% 0.00% 0 HTTP CORE 149 0 1 0 0.00% 0.00% 0.00% 0 RARP Input 150 0 2 0 0.00% 0.00% 0.00% 0 PPP NBF 151 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall 152 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background 153 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana 154 13444 1279784 10 0.00% 0.00% 0.00% 0 QoS stats proces 155 38472 127738943 0 0.00% 0.01% 0.00% 0 RBSCP Background 156 0 2 0 0.00% 0.00% 0.00% 0 SCTP Main Proces 157 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage 3 42564 606271 70 0.00% 0.00% 0.00% 0 VPDN failure man 159 6024 113669 52 0.00% 0.00% 0.00% 0 ADJ background 160 0 1 0 0.00% 0.00% 0.00% 0 ac_atm_state_eve 161 0 3 0 0.00% 0.00% 0.00% 0 gk process 82 95392 541386 176 0.00% 0.00% 0.00% 0 AAA Server 1 168 2294 73 0.00% 0.00% 0.00% 0 Chunk Manager 164 0 1 0 0.00% 0.00% 0.00% 0 Probe Input 165 0 2 0 0.00% 0.00% 0.00% 0 IP Inband Sessio 166 316 3124376 0 0.00% 0.00% 0.00% 0 IP SIP Process 116 66404 397922672 0 0.00% 0.02% 0.02% 0 IPAM Manager 168 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA 169 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr 170 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 171 22216 120536 184 0.00% 0.00% 0.00% 0 LOCAL AAA 172 2664 12495250 0 0.00% 0.00% 0.00% 0 BGP Scheduler 173 0 2 0 0.00% 0.00% 0.00% 0 TPLUS 174 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support 175 0 1 0 0.00% 0.00% 0.00% 0 IDMGR CORE 176 0 5 0 0.00% 0.00% 0.00% 0 MPLS Auto Mesh P 177 0 1 0 0.00% 0.00% 0.00% 0 VOIP RTP Udp Inp 178 0 1 0 0.00% 0.00% 0.00% 0 CCSWVOICE 179 0 1 0 0.00% 0.00% 0.00% 0 QOS_MODULE_MAIN 180 0 12 0 0.00% 0.00% 0.00% 0 crypto engine pr 181 0 4 0 0.00% 0.00% 0.00% 0 Crypto CA 182 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL 183 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL 184 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc 185 0 2 0 0.00% 0.00% 0.00% 0 Proxy Session Ap 186 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_HTSP 187 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_R2 188 0 1 0 0.00% 0.00% 0.00% 0 TSP 189 0 1 0 0.00% 0.00% 0.00% 0 Key Proc 190 0 6 0 0.00% 0.00% 0.00% 0 Crypto ACL 191 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT 192 0 3 0 0.00% 0.00% 0.00% 0 Crypto IKMP 193 672 639853 1 0.00% 0.00% 0.00% 0 IPSEC key engine 194 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key 195 0 1 0 0.00% 0.00% 0.00% 0 Crypto PAS Proc 196 0 1 0 0.00% 0.00% 0.00% 0 GDOI GM Process 197 46004 569599 80 0.00% 0.00% 0.00% 0 AAA SEND STOP EV 198 4 2 2000 0.00% 0.00% 0.00% 0 EEM ED Resource 199 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Track 200 448 1279684 0 0.00% 0.00% 0.00% 0 RMON Recycle Pro 201 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se 202 808 267 3026 0.00% 0.00% 0.00% 0 Syslog Traps 203 0 1 0 0.00% 0.00% 0.00% 0 SYSMGT Events 204 0 1 0 0.00% 0.00% 0.00% 0 DATA Transfer Pr 205 0 1 0 0.00% 0.00% 0.00% 0 DATA Collector 206 0 1 0 0.00% 0.00% 0.00% 0 SONET Traps 207 60 80961 0 0.00% 0.00% 0.00% 0 DFS flush period 208 0 36 0 0.00% 0.00% 0.00% 0 EEM Server 209 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc 210 0 3 0 0.00% 0.00% 0.00% 0 EEM ED CLI 211 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Config 212 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Counter 213 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Env 214 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Interface 215 0 3 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD 216 0 3 0 0.00% 0.00% 0.00% 0 EEM ED None 217 0 3 0 0.00% 0.00% 0.00% 0 EEM ED OIR 218 36 5 7200 0.00% 0.00% 0.00% 0 crypto sw pk pro 219 0 3 0 0.00% 0.00% 0.00% 0 EEM ED SNMP 220 4 3 1333 0.00% 0.00% 0.00% 0 EEM ED Test 221 52 320417 0 0.00% 0.00% 0.00% 0 EEM ED Timer 222 16 289 55 0.00% 0.00% 0.00% 0 Syslog 223 0 1 0 0.00% 0.00% 0.00% 0 IP SLAs Ethernet 224 0 1 0 0.00% 0.00% 0.00% 0 IP SLA AutoEther 225 0 1 0 0.00% 0.00% 0.00% 0 RMON Packets 226 13896 12787511 1 0.00% 0.00% 0.00% 0 trunk conditioni 227 0 1 0 0.00% 0.00% 0.00% 0 trunk conditioni PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 228 600 635704 0 0.00% 0.00% 0.00% 0 IP SLA MPLSLM Pr 229 72 3824 18 0.00% 0.00% 0.00% 0 Net Input 119 0 3 0 0.00% 0.00% 0.00% 0 MPLS Auto-Tunnel 231 24 106647 0 0.00% 0.00% 0.00% 0 CC-API_VCM 232 53032 127739511 0 0.00% 0.02% 0.00% 0 CCPROXY_CT 233 0 1 0 0.00% 0.00% 0.00% 0 CCPROXY_UDP 234 36 38 947 0.00% 0.00% 0.00% 2 Virtual Exec 235 284404 398639241 0 0.00% 0.03% 0.02% 0 RADIUS 236 1040 28923 35 0.00% 0.00% 0.00% 0 Call Progress No 120 0 1 0 0.00% 0.00% 0.00% 0 O-UNI Client Msg 131 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP 239 979568 1684297 581 0.00% 0.00% 0.00% 0 L2TUN Applicatio 240 0 1 0 0.00% 0.00% 0.00% 0 VPDN Test 241 0 2 0 0.00% 0.00% 0.00% 0 LSP Tunnels 242 4752 1083402 4 0.00% 0.00% 0.00% 0 LSP Tunnel Head 243 2384 513204 4 0.00% 0.00% 0.00% 0 IPv6 IDB 244 0 3 0 0.00% 0.00% 0.00% 0 IPv6 RIB Redistr 245 18184 51189283 0 0.00% 0.00% 0.00% 0 MLD 246 164 213299 0 0.00% 0.00% 0.00% 0 IPv6 Input 247 0 1 0 0.00% 0.00% 0.00% 0 IPv6 ND 248 122476 3950812 31 0.00% 0.00% 0.00% 0 CLNS Input 249 37864 2254401 16 0.00% 0.00% 0.00% 0 ES-IS Routing 250 71704 2556732 28 0.00% 0.00% 0.00% 0 LSD Main Proc 251 288204 1279442 225 0.00% 0.00% 0.00% 0 MFI LFD Stats Pr 252 796 853124 0 0.00% 0.00% 0.00% 0 MFI LFD Timer Pr 253 0 2 0 0.00% 0.00% 0.00% 0 MFI LFD Main Pro 254 126828 399905393 0 0.00% 0.06% 0.07% 0 PPP manager 255 0 2 0 0.00% 0.00% 0.00% 0 LSP Verification 256 65604 12782720 5 0.00% 0.00% 0.00% 0 MPLS HC Counter 257 600 213290 2 0.00% 0.00% 0.00% 0 LDP Background 258 27052 2280027 11 0.00% 0.00% 0.00% 0 Tag Control 259 68 213293 0 0.00% 0.00% 0.00% 0 IPRM 145 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder 261 2184 12795268 0 0.00% 0.00% 0.00% 0 Multilink PPP 262 0 2 0 0.00% 0.00% 0.00% 0 Multilink event 263 96628 1760899 54 0.00% 0.00% 0.00% 0 LDP 264 481740 11684424 41 0.00% 0.00% 0.00% 0 TDP Hello 265 728616 6172853 118 0.00% 0.00% 0.00% 0 ISIS Adj 266 128240 2298540 55 0.00% 0.00% 0.00% 0 ISIS Upd 267 89516 2899457 30 0.00% 0.00% 0.00% 0 BGP I/O 268 0 7 0 0.00% 0.00% 0.00% 0 TCP Listener 269 2399580 24686023 97 0.00% 0.00% 0.00% 0 IP SNMP 270 490076 12012732 40 0.00% 0.00% 0.00% 0 PDU DISPATCHER 271 4390468 12013813 365 0.00% 0.00% 0.00% 0 SNMP ENGINE 272 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6 273 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro 274 1722996 625015 2756 0.00% 0.01% 0.00% 0 SNMP Traps 275 0 1 0 0.00% 0.00% 0.00% 0 PW-MIB notifs 276 1552 57726 26 0.00% 0.00% 0.00% 0 MPLS MIB traps 277 96648 368111 262 0.00% 0.00% 0.00% 0 RADIUS EXTENDED 278 4336 12955883 0 0.00% 0.00% 0.00% 0 NTP 279 0 1 0 0.00% 0.00% 0.00% 0 xcpa-driver 280 759840 44336 17138 0.00% 0.07% 0.02% 0 VTEMPLATE Backgr 281 3604 135843 26 0.00% 0.00% 0.00% 0 Tagcon Addr 282 0 4 0 0.00% 0.00% 0.00% 0 IPRM Collection 283 337764 1559802 216 0.00% 0.00% 0.00% 0 BGP Router 284 7253128 854870 8484 0.00% 0.02% 0.00% 0 BGP Scanner PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 285 26344 2996 8793 0.00% 0.00% 0.00% 0 BGP Event 286 144 496 290 0.00% 0.00% 0.00% 3 Virtual Exec 287 12 75 160 0.00% 0.00% 0.00% 4 Virtual Exec Really, nothing really special. As for the TAC case, I am not able to do so. Thanks for the feedbacks. Best regards. Y. 2010/4/22 Yap Chin Hoong - <[email protected]> > Hi Youssef, > > Try to capture the output of the show proc cpu sorted during high CPU, > let's see whether the problem is caused by interrupts or processes. Will you > be able to open a Cisco TAC case and request the TAC engineer to perform a > CPU profiling regarding this? > > regards, > > YapCH > > http://itcertguides.blogspot.com/ > > ------------------------------ > Date: Thu, 22 Apr 2010 15:49:33 +0200 > Subject: Re: [c-nsp] Cisco 7204VXR with NPE-G1 high CPU and output drops > From: [email protected] > To: [email protected] > CC: [email protected] > > > Hello, > > Really, nothing related to the CPU : > > LNS1.IX1#sh processes cpu history > > 2222222223333322222444444444455555333332222222222222222222 > 6666666666666699999444449999911111666666666688888888888888 > 100 > 90 > 80 > 70 > 60 > 50 ********** > 40 ***** ******************** > 30 ********************************************************** > 20 ********************************************************** > 10 ********************************************************** > 0....5....1....1....2....2....3....3....4....4....5....5.... > 0 5 0 5 0 5 0 5 0 5 > CPU% per second (last 60 seconds) > > 4434344555655555455455555454555555555555555555555555565555 > 8253536886175400933900221909246456366223331226663324503316 > 100 > 90 > 80 > 70 > 60 ****** * ** ** *** ** * > 50 * **###########*####################################*# > 40 *******#################################################*# > 30 ########################################################## > 20 ########################################################## > 10 ########################################################## > 0....5....1....1....2....2....3....3....4....4....5....5.... > 0 5 0 5 0 5 0 5 0 5 > CPU% per minute (last 60 minutes) > * = maximum CPU% # = average CPU% > > 6567544433244568899987786666543422242347557976575555443422242335455655 > 1917692522895458476603000342305575449131462755826358236305308599538984 > 100 *** * > 90 * *** * > 80 * ****** * ** > 70 * ***###**** * **** * * > 60 ***** *#######****** * ******** ** * *** > 50 ####** * ***##############* * ************* ******* > 40 ######** **################*** * *######*********** * ***####** > 30 ########****#################***** ***##############**** * ***######## > 20 #################################*######################***########### > 10 ###################################################################### > 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7. > 0 5 0 5 0 5 0 5 0 5 0 5 0 > CPU% per hour (last 72 hours) > * = maximum CPU% # = average CPU% > > > LNS1.IX1#sh processes cpu sorted | e 0.00 > CPU utilization for five seconds: 32%/28%; one minute: 37%; five minutes: > 36% > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process > 167 148648548 669722691 221 1.11% 0.94% 0.71% 0 LFDp Input > Proc > 92 84501540 658096034 128 0.55% 0.48% 0.38% 0 IP > Input > 158 29882772 19491105 1533 0.47% 0.50% 0.50% 0 CEF: IPv4 > proces > 44 21898608 2506816 8735 0.31% 0.16% 0.17% 0 Net > Background > 163 1811203169702554 0 0.23% 0.15% 0.16% 0 HQF Input > Shaper > 230 13727516 2560015 5362 0.23% 0.19% 0.18% 0 Compute load > avg > 238 4050992 17914481 226 0.15% 0.04% 0.05% 0 L2TP mgmt > daemon > 237 3963624 303695077 13 0.15% 0.14% 0.15% 0 L2X Data > Daemon > 260 1429340 404591201 3 0.15% 0.08% 0.08% 0 PPP > Events > 162 2102403169697698 0 0.15% 0.28% 0.27% 0 HQF Shaper > Backg > 234 10364 4211 2461 0.07% 0.08% 0.03% 2 Virtual > Exec > 84 61120 398983803 0 0.07% 0.03% 0.02% 0 ACCT > Periodic Pr > 254 126412 399597741 0 0.07% 0.06% 0.07% 0 PPP > manager > > > > I have been digging around in the archives, looks like I'm victim of > microburst trafic : > > http://puck.nether.net/pipermail/cisco-nsp/2009-April/060158.html > > > What can one do about it exept buy bigger / faster boxes with no garranty > it would work !?! > > Regards. > > Y. > > > > 2010/4/22 Yap Chin Hoong - <[email protected]> > > > Hi Youssef, kindly provide the output of the 'show proc cpu sorted' and > 'show proc cpu history'. Thanks. > regards,YapCHhttp://itcertguides.blogspot.com/ > > Date: Thu, 22 Apr 2010 15:06:11 +0200 > > From: Youssef Bengelloun-Zahr <[email protected]> > > To: [email protected] > > Subject: [c-nsp] Cisco 7204VXR with NPE-G1 high CPU and output drops > > Message-ID: > > <[email protected]> > > Content-Type: text/plain; charset=windows-1252 > > > > Hello community, > > > > I Have a Cisco 7204VXR router with NPE-G1 that started acting weird for > 24 > > hours. This router is dual attached to two 6k5 routers using multimode FO > > and SX GBICs. > > > > The router is used as an LNS termination point for PPPoVPDN sessions, we > > have a bunch of them. > > > > Here is a show ver output : > > > > LNS1.IX1#sh version > > Cisco IOS Software, 7200 Software (C7200-ADVENTERPRISEK9-M), Version > > 12.2(33)SRD, RELEASE SOFTWARE (fc2) > > Technical Support: http://www.cisco.com/techsupport > > Copyright (c) 1986-2008 by Cisco Systems, Inc. > > Compiled Thu 23-Oct-08 12:58 by prod_rel_team > > > > ROM: System Bootstrap, Version 12.3(4r)T1, RELEASE SOFTWARE (fc1) > > BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.3(5a), RELEASE > SOFTWARE > > (fc1) > > > > LNS1.IX1 uptime is 21 weeks, 23 hours, 22 minutes > > System returned to ROM by reload at 13:36:53 UTC Wed Nov 25 2009 > > System restarted at 13:39:45 UTC Wed Nov 25 2009 > > System image file is "disk0:c7200-adventerprisek9-mz.122-33.SRD.bin" > > Last reload type: Normal Reload > > Last reload reason: Reload command > > > > > > > > This product contains cryptographic features and is subject to United > > States and local country laws governing import, export, transfer and > > use. Delivery of Cisco cryptographic products does not imply > > third-party authority to import, export, distribute or use encryption. > > Importers, exporters, distributors and users are responsible for > > compliance with U.S. and local country laws. By using this product you > > agree to comply with applicable laws and regulations. If you are unable > > to comply with U.S. and local laws, return this product immediately. > > > > A summary of U.S. laws governing Cisco cryptographic products may be > found > > at: > > http://www.cisco.com/wwl/export/crypto/tool/stqrg.html > > > > If you require further assistance please contact us by sending email to > > [email protected]. > > > > Cisco 7204VXR (NPE-G1) processor (revision B) with 983040K/65536K bytes > of > > memory. > > Processor board ID 29498611 > > SB-1 CPU at 700Mhz, Implementation 0x401, Rev 0.2, 512KB L2 Cache > > 4 slot VXR midplane, Version 2.7 > > > > Last reset from power-on > > > > PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points. > > Current configuration on bus mb1 has a total of 0 bandwidth points. > > This configuration is within the PCI bus capacity and is supported. > > > > PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points. > > Current configuration on bus mb2 has a total of 0 bandwidth points. > > This configuration is within the PCI bus capacity and is supported. > > > > Please refer to the following document "Cisco 7200 Series Port Adaptor > > Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com> > > for c7200 bandwidth points oversubscription and usage guidelines. > > > > > > 1 FastEthernet interface > > 3 Gigabit Ethernet interfaces > > 509K bytes of NVRAM. > > > > 1000944K bytes of ATA PCMCIA card at slot 0 (Sector size 512 bytes). > > 1000944K bytes of ATA PCMCIA card at slot 1 (Sector size 512 bytes). > > 62592K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes). > > 16384K bytes of Flash internal SIMM (Sector size 256K). > > Configuration register is 0x2102 > > > > > > Starting yesterday afternoon, I saw appear high CPU usage and numerous > > output drops. My first instinct was that GBICs started dying so I > replaced, > > no change. > > > > Then, I thought we were victim of a DDoS but my graphs show no increase > of > > number of packets or things like that. > > > > I have been debugging this and found out this : > > > > LNS1.IX1#sh interfaces gi0/2 controller > > GigabitEthernet0/2 is up, line protocol is up > > Hardware is BCM1250 Internal MAC, address is 000b.fcdd.c41a (bia > > 000b.fcdd.c41a) > > Description: F=B, E=BB2.IX1, P=Gi9/1 > > Internet address is 77.246.80.101/31 > > MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec, > > reliability 255/255, txload 29/255, rxload 34/255 > > Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set > > Keepalive set (10 sec) > > Full Duplex, 1000Mbps, link type is auto, media type is SX > > output flow-control is XON, input flow-control is XON > > ARP type: ARPA, ARP Timeout 04:00:00 > > Last input 00:00:00, output 00:00:00, output hang never > > Last clearing of "show interface" counters 00:22:35 > > Input queue: 1/150/0/0 (size/max/drops/flushes); Total output drops: > 2712 > > *Queueing strategy: Class-based queueing* > > Output queue: 311/1000/0 (size/max total/drops) > > 5 minute input rate 133959000 bits/sec, 28885 packets/sec > > 5 minute output rate 115013000 bits/sec, 20137 packets/sec > > 39283651 packets input, 1536817418 bytes, 0 no buffer > > Received 535 broadcasts (0 IP multicasts) > > 0 runts, 0 giants, 0 throttles > > 164 input errors, 0 CRC, 0 frame, 164 overrun, 0 ignored > > 0 watchdog, 539 multicast, 0 pause input > > 27456129 packets output, 2625296538 bytes, 0 underruns > > 0 output errors, 0 collisions, 0 interface resets > > 0 babbles, 0 late collision, 0 deferred > > 0 lost carrier, 0 no carrier, 0 pause output > > 0 output buffer failures, 0 output buffers swapped out > > Interface GigabitEthernet0/2 (idb 0x50098218) > > Hardware is BCM1250 Internal MAC (Revision B2/B3) > > Network connection mode is AUTO > > network link is up > > Config is 1000Mbps, Full Duplex > > Selected media-type is GBIC > > GBIC type is 1000BaseSX > > MAC Registers: > > mac_cfg = 0x000000C8000A0176, mac_thrsh_cfg = > 0x0000080400084004 > > mac_vlantag = 0x0000000000000000, mac_frame_cfg = > 0x241C400000280200 > > mac_adfilter_cfg = 0x0000000000000E28, mac_enable = > 0x0000000000000C11 > > mac_status = 0x0000000000040004, mac_int_mask = > 0x00004F0000C300C3 > > mac_txd_ctl = 0x000000000000000F, mac_eth_addr = > 0x00001AC4DDFC0B00 > > mac_fifo_ptrs = 0x241C400000280200, mac_eopcnt = > 0x000044001B1B1B1B > > MAC RX is enabled RX DMA - channel 0 is enabled, channel 1 is disabled > > MAC TX is enabled TX DMA - channel 0 is enabled, channel 1 is disabled > > Device status = 1000 Mbps, Full-Duplex > > PHY Registers: > > PHY is Marvell 88E1011S (Rev 1.3) > > Control = 0x1000 Status = > 0x796D > > PHY ID 1 = 0x0141 PHY ID 2 = > 0x0C62 > > Auto Neg Advertisement = 0x01A0 Link Partner Ability = > 0x4120 > > Auto Neg Expansion = 0x0000 Next Page Tx = > 0x2001 > > Link Partner Next Page = 0x0000 1000BaseT Control = > 0x0000 > > 1000BaseT Status = 0x0000 Extended Status = > 0xC000 > > PHY Specific Control = 0x0008 PHY Specific Status = > 0xAD04 > > Interrupt Enable = 0x6C00 Interrupt Status = > 0x0000 > > Ext PHY Spec Control = 0x0C64 Receive Error Counter = > 0x0000 > > LED Control = 0x4100 > > Ext PHY Spec Control 2 = 0x006A Ext PHY Spec Status = > 0xA017 > > PHY says Link is UP, Speed 1000Mbps, Full-Duplex [AUTONEG Done] > > Physical Interface - GBIC > > AUTONEG - Our ability is 1000M/FD Pause Capable (Asymmetric) > > AUTONEG - Partner ability is 1000M/FD > > GBIC registers: > > Register 0x00: 01 04 01 00 00 00 01 20 > > Register 0x08: 40 0C 01 01 0D 00 00 00 > > Register 0x10: 37 1E 00 00 4F 45 4D 20 > > Register 0x18: 20 20 20 20 20 20 20 20 > > Register 0x20: 20 20 20 20 00 00 00 00 > > Register 0x28: 47 42 49 43 2D 53 58 20 > > Register 0x30: 20 20 20 20 20 20 20 20 > > Register 0x38: 00 00 00 00 03 52 00 BA > > Register 0x40: 00 1A 00 00 42 31 30 34 > > Register 0x48: 38 31 39 31 20 20 20 20 > > Register 0x50: 20 20 20 20 30 39 30 39 > > Register 0x58: 32 39 20 20 68 B0 01 5A > > Register 0x60: 20 20 20 20 20 20 20 20 > > Register 0x68: 20 20 20 20 20 20 20 20 > > Register 0x70: 20 20 20 20 20 20 20 20 > > Register 0x78: 20 20 20 20 20 20 20 20 > > PartNumber: GBIC-SX > > PartRev: B > > SerialNo: B1048191 > > Options: 0 > > Length(9um/50um/62.5um): 000/550/300 > > Date Code: 090929 > > Gigabit Ethernet Codes: 1 > > Internal Driver Information: > > lc_ip_turbo_fs = 0x6236BE18, ip_routecache = 0x11 (dfs = 0/mdfs = 0) > > rx cache size = 1000, rx cache end = 15 > > max_mtu = 9244 > > Software MAC address filter(hash:length/addr/mask/hits): > > need_af_check = 0 > > 0x00: 0 ffff.ffff.ffff 0000.0000.0000 0 > > 0x2E: 0 0900.2b00.0005 0000.0000.0000 0 > > 0x2F: 0 0900.2b00.0004 0000.0000.0000 0 > > 0x5C: 0 0100.5e00.0002 0000.0000.0000 0 > > 0xC0: 0 0100.0ccc.cccc 0000.0000.0000 0 > > 0xD6: 0 0180.c200.0014 0000.0000.0000 0 > > 0xD7: 0 0180.c200.0015 0000.0000.0000 0 > > 0xE6: 0 000b.fcdd.c41a 0000.0000.0000 0 > > ring sizes: RX = 128, TX = 256 > > rx_particle_size: 512 > > Rx Channel 0: > > dma_config0 = 0x0010002000800888, dma_config1 = > 0x002D000000600029 > > dma_dscr_base = 0x000000000C218A40, dma_dscr_cnt = > 0x0000000000000080 > > dma_cur_dscr_a = 0x000010000C29FC82, dma_cur_dscr_b = > 0x02D4000000000001 > > dma_cur_daddr = 0x000080000C2190E0 > > rxring = 0x0C218A40, shadow = 0x50098FA4, head = 26 (0x0C218BE0) > > rx_overrun=78512, rx_nobuffer=0, rx_discard=0 > > Error Interrupts: rx_int_dscr = 0, rx_int_derr = 0, rx_int_drop = 53 > > Tx Channel 0: > > dma_config0 = 0x0000000001001088, dma_config1 = > 0x00B6000000000010 > > dma_dscr_base = 0x000000000C219280, dma_dscr_cnt = > 0x0000000000000000 > > dma_cur_dscr_a = 0x00000F000C27F980, dma_cur_dscr_b = > 0x0000000000000000 > > dma_cur_daddr = 0x000000000C219430 > > txring = 0x0C219280, shadow = 0x657D0A14, head = 164, tail = 165, > tx_count > > = 1 > > Error Interrupts: tx_int_dscr = 0, tx_int_derr = 0, tx_int_dzero = 0 > > chip_state = 2, ds->tx_limited = 0 > > throttled = 0, enabled = 0, disabled = 0 > > reset=6(init=1, restart=5), auto_restart=1 > > tx_underflow = 0, tx_overflow = 0 > > rx_underflow = 0, rx_overflow = 0, filtered_pak=0 > > descriptor mismatch = 0, fixed alignment = 52530 > > bad length = 0 dropped, 0 corrected > > unexpected sop = 0 > > Address Filter: > > Promiscuous mode OFF > > Exact match table (for unicast, maximum 8 entries): > > Entry 0 MAC Addr = 000b.fcdd.c41a > > (All other entries are empty) > > Hash match table (for multicast, maximum 8 entries): > > Entry 0 MAC Addr = 0100.0ccc.cccc > > Entry 1 MAC Addr = 0900.2b00.0004 > > Entry 2 MAC Addr = 0900.2b00.0005 > > Entry 3 MAC Addr = 0180.c200.0014 > > Entry 4 MAC Addr = 0180.c200.0015 > > Entry 5 MAC Addr = 0100.5e00.0002 > > (All other entries are empty) > > Statistics: > > Rx Bytes 19767582054 Tx Bytes > > 19944011802 > > Rx Good Packets 27498925 Tx Good Packets > > 27480126 > > Rx Multicast 544 > > Rx Broadcast 0 > > > > Rx Bad Pkt Errors 0 Tx Bad Pkt Errors > > 0 > > Rx FCS Errors 0 Tx FCS Errors > > 0 > > Rx Runt Errors 0 Tx Runt Errors > > 0 > > Rx Oversize Errors 0 Tx Oversize Errors > > 0 > > Rx Length Errors 0 Tx Collisions > > 0 > > Rx Code Errors 0 Tx Late Collisions > > 0 > > Rx Dribble Errors 0 Tx Excessive Collisions > > 0 > > Tx Abort Errors > > 0 > > > > > > My queuing strategy went from FIFO to Class-based queuing !?! How is that > > possible ? > > > > Any ideas on what might be causing this ? > > > > Thanks. > > > > Regards. > > > > Y. > > > > -- > > Youssef BENGELLOUN-ZAHR ?????????????????? > > Ing?nieur R?seaux et T?l?coms > > > > > > Technopole de l'Aube en Champagne - BP 601 - 10901 TROYES Cedex 9 > > Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE > > Tel +33 (0) 825 000 720 > > Tel. direct +33 (0) 1 77 35 59 14 > > Tel. portable +33 (0) 6 22 42 63 80 > > Email [email protected] > > ??????????????????????????????.....www.720.fr > > > > _________________________________________________________________ > The New Busy is not the too busy. Combine all your e-mail accounts with > Hotmail. > > http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4 > _______________________________________________ > cisco-nsp mailing list [email protected] > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > > > > > -- > Youssef BENGELLOUN-ZAHR ……………………………………………… > Ingénieur Réseaux et Télécoms > > > Technopole de l'Aube en Champagne - BP 601 - 10901 TROYES Cedex 9 > Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE > Tel +33 (0) 825 000 720 > Tel. direct +33 (0) 1 77 35 59 14 > Tel. portable +33 (0) 6 22 42 63 80 > Email [email protected] > ……………………………………………………………………………….....www.720.fr > > > ------------------------------ > Hotmail is redefining busy with tools for the New Busy. Get more from your > inbox. See > how.<http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2> > -- Youssef BENGELLOUN-ZAHR ……………………………………………… Ingénieur Réseaux et Télécoms Technopole de l'Aube en Champagne - BP 601 - 10901 TROYES Cedex 9 Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE Tel +33 (0) 825 000 720 Tel. direct +33 (0) 1 77 35 59 14 Tel. portable +33 (0) 6 22 42 63 80 Email [email protected] ……………………………………………………………………………….....www.720.fr _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
