That 12.4(3) IOS is pretty old. Trying a newer one might help, as you're vulnerable to many things. It's possible there are bugs you're hitting that are affecting performance. If you could consolidate some things, that may help. You're matching RTP, but also matching packet length, that might be overkill. The fast hellos for OSPF probably aren't helping either. Another thought might be to score a 2950 or 3550 L2 switch, and put that in place of the 2924. Then move all the ACls to that, as it can do them in hardware. You could probably do a little buffer tuning, middle ones look pretty ugly. Probably not long term solution. I think MCQ is more efficient than CAR, might want to move to that completely.
Chuck -----Original Message----- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Robert Johnson Sent: Tuesday, March 24, 2009 10:55 AM To: cisco-nsp@puck.nether.net Subject: [c-nsp] OSPF and iBGP session drops between 3640s Hello list, I have a small network with four 3640s. Each router has 128/32MB ram, and a single FE interface connected to a catalyst 2924. Two of the routers are running BGP, each with a session to a (single) other provider, and a session between themselves. These are not carrying full tables. All four routers are running OSPF between each other. The problem is that occasionally (from once a week to 3x/day) OSPF neighbor relationships will bounce due to hello timers expiring. Just recently the iBGP session between two of the routers also bounced. There do not appear to be any layer 1 or 2 connectivity problems that would cause this behavior. However, CPU usage on the 3640s seems high- 30% sustained and up to 90% peak, with only 1-2k max PPS. Also, I'm seeing buffer misses and failures. CEF is enabled. There are several relatively long access lists that are being processed, and the routers are doing QoS classifying and tagging at layers 2 and 3 for VoIP performance. Without any major hardware changes, where do I begin here? Thanks in advance. The fun stuff (sho buffers, sho proc cpu hist, sho proc cpu, sho run): router1#sho buffers Buffer elements: 1118 in free list (500 max allowed) 707983613 hits, 0 misses, 1119 created Public buffer pools: Small buffers, 104 bytes (total 78, permanent 50, peak 104 @ 4w0d): 42 in free list (20 min, 150 max allowed) 18990955 hits, 3598 misses, 4408 trims, 4436 created 312 failures (0 no memory) Middle buffers, 600 bytes (total 25, permanent 25, peak 176 @ 7w0d): 22 in free list (10 min, 150 max allowed) 651012877 hits, 12602 misses, 30744 trims, 30744 created 2744 failures (0 no memory) Big buffers, 1536 bytes (total 50, permanent 50, peak 63 @ 2d19h): 50 in free list (5 min, 150 max allowed) 4658228 hits, 1005 misses, 102 trims, 102 created 936 failures (0 no memory) VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 12 @ 7w0d): 10 in free list (0 min, 100 max allowed) 129 hits, 807 misses, 13 trims, 13 created 807 failures (0 no memory) Large buffers, 5024 bytes (total 1, permanent 0, peak 3 @ 7w0d): 1 in free list (0 min, 10 max allowed) 14 hits, 793 misses, 2764 trims, 2765 created 793 failures (0 no memory) Huge buffers, 18024 bytes (total 1, permanent 0, peak 3 @ 7w0d): 1 in free list (0 min, 4 max allowed) 16 hits, 779 misses, 3858 trims, 3859 created 778 failures (0 no memory) Interface buffer pools: CD2430 I/O buffers, 1536 bytes (total 0, permanent 0): 0 in free list (0 min, 0 max allowed) 0 hits, 0 fallbacks Header pools: Header buffers, 0 bytes (total 265, permanent 256, peak 265 @ 7w0d): 9 in free list (10 min, 512 max allowed) 253 hits, 3 misses, 0 trims, 9 created 0 failures (0 no memory) 256 max cache size, 256 in cache 7674266 hits in cache, 0 misses in cache Particle Clones: 1024 clones, 0 hits, 0 misses Public particle pools: F/S buffers, 256 bytes (total 384, permanent 384): 128 in free list (128 min, 1024 max allowed) 256 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 256 max cache size, 256 in cache 0 hits in cache, 0 misses in cache Normal buffers, 1548 bytes (total 512, permanent 512): 384 in free list (128 min, 1024 max allowed) 21114 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 128 max cache size, 128 in cache 0 hits in cache, 0 misses in cache Private particle pools: IDS SM buffers, 240 bytes (total 128, permanent 128): 0 in free list (0 min, 128 max allowed) 128 hits, 0 fallbacks 128 max cache size, 128 in cache 0 hits in cache, 0 misses in cache FastEthernet0/0 buffers, 1548 bytes (total 192, permanent 192): 0 in free list (0 min, 192 max allowed) 192 hits, 0 fallbacks 192 max cache size, 128 in cache 694772430 hits in cache, 20986 misses in cache router1#sho proc cpu hist router1 02:40:53 PM Tuesday Mar 24 2009 UTC 4444444444444444444444444444444555554444444444444443333355 8333332222200000000004444411111111110000000000222227777722 100 90 80 70 60 50 * ***** **** 40 ************************************************************ 30 ************************************************************ 20 ************************************************************ 10 ************************************************************ 0....5....1....1....2....2....3....3....4....4....5....5.... 0 5 0 5 0 5 0 5 0 5 CPU% per second (last 60 seconds) 5656435544454334664445454566532243344446444645545774454545 2900663259495363238448467911347711166900544033873220265057 100 90 80 70 * ** 60 * * * ** * * *** * * * ** * * 50 ***#* *#** ** *** * **###* *** ** * ****#* ******* 40 ##*##*####*#* **####*#***###* * **********######****##### 30 ##############################**#***##*#**################## 20 ############################################################ 10 ############################################################ 0....5....1....1....2....2....3....3....4....4....5....5.... 0 5 0 5 0 5 0 5 0 5 CPU% per minute (last 60 minutes) * = maximum CPU% # = average CPU% 7664665666756555557666776555554545664455555555455555654555556565545654 2734555279005332498657259890379052808981353640965081868135475217086638 100 90 80 * * 70 ** ** *** * ******* * * * * * 60 *** ******* * ********** * ** * * * ** * ** * ** ** ** ** 50 *** ******************************************************************** 40 ****##########****#***##****########################*################### 30 ##**###########***########**############################################ 20 ###*############*##########*############################################ 10 ######################################################################## 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7. 0 5 0 5 0 5 0 5 0 5 0 5 0 CPU% per hour (last 72 hours) * = maximum CPU% # = average CPU% sho proc cpu: CPU utilization for five seconds: 42%/39%; one minute: 43%; five minutes: 40% router1#sho run Building configuration... Current configuration : 8460 bytes ! ! Last configuration change at 01:54:37 UTC Tue Mar 24 2009 ! NVRAM config last updated at 22:25:51 UTC Thu Mar 5 2009 ! version 12.4 service timestamps debug uptime service timestamps log uptime service password-encryption ! hostname router1 ! boot-start-marker boot system flash c3640-jk9o3s-mz.124-3.bin boot-end-marker ! no logging console enable secret 5 ** ! no aaa new-model ! resource policy ! ip subnet-zero no ip source-route ! ! ip cef ! ! class-map match-all assure match ip dscp af31 class-map match-all critical match ip dscp cs6 class-map match-all expedite match ip dscp ef class-map match-any rtp-vox match ip rtp 13456 13462 match ip rtp 13556 13560 match ip rtp 13656 13660 match ip rtp 13756 13760 class-map match-all sip match protocol sip class-map match-all voice match packet length min 1 max 200 match class-map rtp-vox ! ! policy-map output-cos class expedite set cos 6 class assure set cos 5 class critical set cos 7 policy-map input-mark class sip set ip dscp af31 class voice set dscp ef ! ! ! ! ! ! interface FastEthernet0/0 description Trunk to cat2924-pri no ip address full-duplex ! interface FastEthernet0/0.5 description Switch management segment encapsulation dot1Q 5 ip address 10.1.5.254 255.255.255.0 ip access-group mgmt-only in ip access-group mgmt-only out no snmp trap link-status ! interface FastEthernet0/0.15 description AP management segment encapsulation dot1Q 15 ip address 10.1.15.254 255.255.255.0 ip access-group mgmt-only in ip access-group mgmt-only out no snmp trap link-status ! interface FastEthernet0/0.25 description CTM management segment encapsulation dot1Q 25 ip address 10.1.25.254 255.255.255.0 ip access-group mgmt-only in ip access-group mgmt-only out no snmp trap link-status ! interface FastEthernet0/0.35 description UPS management segment encapsulation dot1Q 35 ip address 10.1.35.254 255.255.255.0 ip access-group mgmt-only in ip access-group mgmt-only out no snmp trap link-status ! interface FastEthernet0/0.50 description Management link to router3 bandwidth 9850 encapsulation dot1Q 50 ip address 10.1.50.254 255.255.255.0 ip access-group mgmt-only in ip access-group mgmt-only out ip ospf message-digest-key 1 md5 7 *secret* ip ospf hello-interval 1 ip ospf dead-interval 5 no snmp trap link-status ! interface FastEthernet0/0.51 description Management link to router2 encapsulation dot1Q 51 ip address 10.1.51.254 255.255.255.0 ip access-group mgmt-only in ip access-group mgmt-only out ip ospf message-digest-key 1 md5 7 ** ip ospf hello-interval 1 ip ospf dead-interval 5 no snmp trap link-status ! interface FastEthernet0/0.52 description Management link to ** bandwidth 10610 encapsulation dot1Q 52 ip address 10.1.52.254 255.255.255.0 ip access-group mgmt-only in ip access-group mgmt-only out no snmp trap link-status ! interface FastEthernet0/0.300 description Production traffic link to router3 bandwidth 9850 encapsulation dot1Q 300 ip address xxx.xxx.xxx.xxx 255.255.255.252 ip ospf message-digest-key 10 md5 7 ** ip ospf dead-interval minimal hello-multiplier 4 no snmp trap link-status service-policy output output-cos ! interface FastEthernet0/0.301 description Production traffic link to router2 encapsulation dot1Q 301 ip address xxx.xxx.xxx.xxx 255.255.255.252 ip ospf message-digest-key 10 md5 7 ** ip ospf dead-interval minimal hello-multiplier 4 no snmp trap link-status service-policy output output-cos ! interface FastEthernet0/0.302 description Production traffic link to ** bandwidth 10610 encapsulation dot1Q 302 ip address xxx.xxx.xxx.xxx 255.255.255.252 ip access-group internet-edge-ingress in ip access-group internet-edge-egress out no snmp trap link-status service-policy input input-mark service-policy output output-cos ! interface FastEthernet0/0.500 description Customer access subnet encapsulation dot1Q 500 ip address xxx.xxx.xxx.xxx 255.255.255.240 ip access-group block-customercrap in ip verify unicast reverse-path rate-limit input access-group 100 768000 10000 200000 conform-action transmit e xceed-action drop rate-limit output access-group 100 768000 40000000 80000000 conform-action tran smit exceed-action drop no snmp trap link-status service-policy output output-cos ! router ospf 1000 log-adjacency-changes area 0.0.0.0 authentication message-digest passive-interface default no passive-interface FastEthernet0/0.300 no passive-interface FastEthernet0/0.301 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0 default-information originate metric-type 1 ! router ospf 100 log-adjacency-changes area 10.0.0.0 authentication message-digest area 10.0.0.0 stub no-summary passive-interface default no passive-interface FastEthernet0/0.50 no passive-interface FastEthernet0/0.51 network 10.0.0.0 0.255.255.255 area 10.0.0.0 ! router bgp ***** no synchronization bgp log-neighbor-changes network xxx.xxx.xxx.xxx mask 255.255.255.192 network xxx.xxx.xxx.xxx mask 255.255.255.192 network xxx.xxx.xxx.xxx mask 255.255.255.192 network xxx.xxx.xxx.xxx mask 255.255.255.192 aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only redistribute ospf 1000 neighbor xxx.xxx.xxx.xxx remote-as **** neighbor xxx.xxx.xxx.xxx route-map pri-map out neighbor xxx.xxx.xxx.xxx remote-as ***** neighbor xxx.xxx.xxx.xxx next-hop-self no auto-summary ! no ip http server no ip http secure-server ip classless ! ! ! ! ip access-list standard mgmt-only permit 10.0.0.0 0.255.255.255 permit 192.168.101.0 0.0.0.255 ! ip access-list extended block-customercrap deny udp any any eq bootps deny tcp any any eq 139 deny tcp any any eq 445 deny udp any any eq netbios-ns deny udp any any eq netbios-dgm permit ip any any ip access-list extended internet-edge-egress deny ip 10.0.0.0 0.255.255.255 any deny ip 172.16.0.0 0.15.255.255 any deny ip 192.168.0.0 0.0.255.255 any deny ip 127.0.0.0 0.0.255.255 any deny ip 224.0.0.0 31.255.255.255 any deny ip 169.254.0.0 0.0.255.255 any deny udp any any eq bootps deny udp any any eq bootpc deny tcp any any eq 139 deny tcp any any eq 445 deny udp any any eq netbios-ns deny udp any any eq netbios-dgm deny ip any xxx.xxx.xxx.xxx 0.0.0.63 deny ip any xxx.xxx.xxx.xxx 0.0.0.63 deny ip any xxx.xxx.xxx.xxx 0.0.0.63 deny ip any xxx.xxx.xxx.xxx 0.0.0.63 permit ip any any ip access-list extended internet-edge-ingress deny ip 10.0.0.0 0.255.255.255 any deny ip 172.16.0.0 0.15.255.255 any deny ip 192.168.0.0 0.0.255.255 any deny ip 127.0.0.0 0.0.255.255 any deny ip 224.0.0.0 31.255.255.255 any deny ip 169.254.0.0 0.0.255.255 any deny udp any any eq bootps deny udp any any eq bootpc deny tcp any any eq 139 deny tcp any any eq 445 deny udp any any eq netbios-ns deny udp any any eq netbios-dgm deny ip xxx.xxx.xxx.xxx 0.0.0.63 any deny ip xxx.xxx.xxx.xxx 0.0.0.63 any deny ip xxx.xxx.xxx.xxx 0.0.0.63 any deny ip xxx.xxx.xxx.xxx 0.0.0.63 any permit ip any any logging facility local5 logging 10.3.40.105 access-list 1 permit xxx.xxx.xxx.xxx 0.0.0.63 access-list 1 permit xxx.xxx.xxx.xxx 0.0.0.63 access-list 2 permit xxx.xxx.xxx.xxx 0.0.0.63 access-list 2 permit xxx.xxx.xxx.xxx 0.0.0.63 access-list 100 permit ip host xxx.xxx.xxx.xxx any access-list 100 permit ip any host xxx.xxx.xxx.xxx snmp-server community ** RO mgmt-only ! route-map pri-map permit 10 match ip address 1 ! route-map pri-map permit 20 match ip address 2 ! ! ! control-plane ! ! ! ! ! ! ! ! ! banner login ^C Property of **. Unauthorized access attempt s will be prosecuted. ^C ! line con 0 password 7 ** login line aux 0 password 7 ** login line vty 0 4 access-class mgmt-only in password 7 ** login ! ntp clock-period 17179597 ntp server 10.3.40.105 ! end _______________________________________________ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ _______________________________________________ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/