Thank you for the reply on this. We did exactly what you mention here
(trying to isolate channels) and found the performance metrics didn't change
very much except that there seemed to be little impairment with just a
single T-1. We do not believe that variance in latency exists to the point
that we should be having a severe issue and it has since reoccurred on a
couple of other bundled connections (on this same particular router - see
below).
None of the T-1s seem to take errors in any of the bundles. We do see a lot
of output queue drops on the Multilink interfaces but not sure how
concerning that really is.
The only difference between this device and similar ones on our network is
that we have exceeded the number of fast interfaces (4 vs. recommended 3 -
but the card in question is in the middle and should be getting its SRAM
allotment okay) and we do terminate some ATM/PPPoE/L2TP sessions on this
device. The system is:
7206 (non-VXR)
NPE-200 with IO-FE
IOS 12.2(31) [bootldr 12.0(13)S]
(is there perhaps an issue in 12.2(31) with MLPPP?
I'd like to go to a 12.3 release but need to verify
Support for the CT3/4T1 for two of our boxes).
We're using the older CT3/4T1 cards on this edge device and haven't had
problems with MLPPP in the past on a similar system (running 12.2(23)c).
Download speed continues to perform okay in most tests but uploads get
woefully bad and we start losing packets above 1.6 to 2.0 mbps (2% observed
today as things crept over 2mbps) regardless of the number of bundled trunks
[2 or 3]. It "seems" that performance improves in the evenings when there
is less traffic going through the device, it's lightly loaded even during
the day (maybe a total of 10 mbps being handled on this one system).
I considered tweaking the buffers, but if it's an issue of emptying the
queues fast enough (perhaps because it's servicing one too many high speed
interfaces?) than putting more in the buffers that it can't get to might
just make things worse.
We have several customers utilizing VoIP and have some policy-maps on those
interfaces, none of them using MLPPP [yet] but a few on the same box and
even the same card in question here. No complaints about lost packets or
voice quality there so the overall system seems sound and CPU utilization is
generally in the low double digits. Various debug outputs don't seem to
barking either.
Any suggestions are appreciated. I think I'm close to just dropping another
chassis in with this DS3 on it and seeing if the problem cleans up.
ADDITIONAL OUTPUTS
7206#show int multilink3
Multilink3 is up, line protocol is up
Hardware is multilink group interface
Internet address is xx.xx.xx.xx/30
MTU 1500 bytes, BW 3072 Kbit, DLY 100000 usec,
reliability 255/255, txload 10/255, rxload 189/255
Encapsulation PPP, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 2 seconds on reset
LCP Open, multilink Open
Open: IPCP
Last input 15:29:57, output never, output hang never
Last clearing of "show interface" counters 20:35:24
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 32796
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 2278000 bits/sec, 236 packets/sec
30 second output rate 131000 bits/sec, 139 packets/sec
7130649 packets input, 312942772 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
6358409 packets output, 2746174328 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitionsLLDC-7206#show buffers failures
7206#show buffers failures
Caller Pool Size When
0x606D1B64 Middle 60 12:13:34
0x606D1B64 Small 62 10:51:14
0x606D1B64 Small 62 10:51:14
0x606D1B64 Small 62 10:51:14
0x606D1B64 Small 62 10:51:14
0x606D1B64 Small 62 05:03:08
0x606D1B64 Small 62 05:03:08
0x606D1B64 Small 62 05:03:08
0x606D1B64 Small 62 05:03:08
0x606D1B64 Small 62 05:03:08
CPU utilization for five seconds: 9%/9%; one minute: 10%; five minutes: 10%
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
7206#show proc | incl IP
11 Mwe 606DA624 0 1 0 5644/6000 0 IPC Zone
Manager
12 Mwe 606DA38C 60 74335 0 5704/6000 0 IPC Periodic
Tim
13 Mwe 606DA330 56 74335 0 5708/6000 0 IPC Deferred
Por
14 Mwe 606DA438 0 1 0 5600/6000 0 IPC Seat
Manager
40 Mwe 60755094 281008 834496 33610332/12000 0 IP Input
45 Mwe 608D4FC4 24 186 129 4884/6000 0 PPP IP Add
Route
49 Mwe 607CD814 1352 16162 83 7328/9000 0 IP
Background
50 Mwe 607D31C8 72 1436 50 7916/9000 0 IP RIB
Update
71 Lsi 60816C44 172 1239 138 5232/6000 0 IP Cache
Ager
115 Lwe 607918C8 0 2 011472/12000 0 IP SNMP
7206#show proc | incl PPP
3 Mwe 608EFC18 1084 433 250321644/24000 0 PPP auth
42 Mwe 6087ED38 0 1 0 5636/6000 0 PPPATM
Session d
45 Mwe 608D4FC4 24 186 129 4884/6000 0 PPP IP Add
Route
102 Mwe 60F15E78 0 1 0 5632/6000 0 PPPOE
discovery
103 Mwe 60F15F48 0 1 0 5624/6000 0 PPPOE
background
110 Mwe 608D5200 34400 196190 17521944/24000 0 PPP manager
111 Hwe 609090AC 976 74368 13 4996/6000 0 Multilink
PPP
112 Hwe 608FF1A0 0 2 0 5576/6000 0 Multilink
PPP ou
7206#show proc | incl Multilink
111 Hwe 609090AC 976 74411 13 4996/6000 0 Multilink
PPP
112 Hwe 608FF1A0 0 2 0 5576/6000 0 Multilink
PPP ou
113 Mwe 609091B0 12 18 666 5060/6000 0 Multilink
event
-----Original Message-----
From: Rodney Dunn [mailto:[EMAIL PROTECTED]
Sent: Saturday, May 26, 2007 8:30 AM
To: Sean Shepard
Cc: cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] Multilink PPP (MLPPP) Asymmetrical Throughput Problem
NxT1
I saw this exact problem from a customer a few months ago. What his
turned out to be was some extra latency on one of the T1 links.
I don't remember if it was in one direction or not though.
How about trying the combinations of 2xT1 bundles to see if you
can isolate one T1 as a particular problem?
Capture 'sh ppp mul' on both sides when you are seeing the
problem and not.
Rodney
On Fri, May 25, 2007 at 10:15:52PM -0400, Sean Shepard wrote:
> I'm encountering an odd problem bonding T1 circuits with Multilink PPP
> (Cisco 7206 down to a 2651XM). In this particular case, I've got three
(3)
> T-1 lines bonded together and, taking into account overhead, I'm getting
> (via direct Ethernet -> laptop connection - no other traffic):
>
> 4.2 to 4.3 mbps downstream, but .
> Only around 2.0 mbps upstream (tried two different "check my speed"
sites).
>
> I was able to initiate multiple large packet repeating pings between the
two
> devices and the show interface statistics showed in the neighborhood of
4.0
> mbps in both directions. This put the CPU in the 60 to 70 percent
> utilization range on the remote 2651XM device (with it originating about
50%
> of the pings but not otherwise routing any traffic, Ethernet interface was
> disconnected) 7206 Utilization (handling traffic for numerous locations)
was
> well under 10%. No ACLs or Policy-Maps at either end of the links.
>
> Any suggestions or insight on the problem are greatly appreciated. I've
not
> observed this kind of difference on various other 2xT1 (3 mbps)
connections
> that I've got in place but this is the first 3xT1. Thanks a bunch!
>
>
> HOST END
> Cisco 7206 (non VXR)
>
> interface Multilink2
> description Multilink (ch16-18)
> ip unnumbered Loopback2
> ppp multilink
> no ppp multilink fragmentation
> multilink-group 2
>
> interface Serial3/0:16
> description MLPPP #1
> mtu 1524
> no ip address
> no ip redirects
> encapsulation ppp
> no fair-queue
> no ppp lcp fast-start
> ppp multilink
> multilink-group 2
> !
> interface Serial3/0:17
> description MLPPP #2
> mtu 1524
> no ip address
> no ip redirects
> encapsulation ppp
> no fair-queue
> ppp multilink
> multilink-group 2
> !
> interface Serial3/0:18
> description MLPPP #3
> mtu 1524
> no ip address
> no ip redirects
> encapsulation ppp
> no fair-queue
> ppp multilink
> multilink-group 2
>
>
>
> REMOTE END
> CISCO 2651XM w/WIC-DSU-T1 Cards
> IOS C2600 Version 12.3(6a)
>
> Router> show run (relevant sections)
>
> network-clock-participate slot 1
> no network-clock-participate wic 0
> no aaa new-model
> ip subnet-zero
> ip cef
> !
> !
> !
> interface Multilink1
> description MultiLink PPP 3xT1
> ip address xx.xx.xx.xx xx.xx.xx.xx
> no ip mroute-cache
> load-interval 30
> ppp multilink
> ppp multilink fragment disable
> ppp multilink group 1
> !
> interface FastEthernet0/0
> description LAN Interface
> ip address xx.xx.xx.xx 255.255.255.248
> speed auto
> full-duplex
> no cdp enable
> !
> interface Serial0/0
> description MLPPP T1 1
> mtu 1524
> no ip address
> encapsulation ppp
> load-interval 120
> no fair-queue
> ppp multilink
> ppp multilink group 1
> ppp multilink endpoint none
> !
> interface Serial0/1
> description MLPPP T1 2
> mtu 1524
> no ip address
> encapsulation ppp
> load-interval 120
> no fair-queue
> ppp multilink
> ppp multilink group 1
> ppp multilink endpoint none
> !
> interface Serial1/0
> description MLPPP T1 3
> mtu 1524
> no ip address
> encapsulation ppp
> load-interval 120
> no fair-queue
> ppp multilink
> ppp multilink group 1
> ppp multilink endpoint none
> !
>
>
>
> ROUTER> show interface
>
> Serial0/0 is up, line protocol is up
> Hardware is PQUICC with Fractional T1 CSU/DSU
> Description: MLPPP T1 1
> MTU 1524 bytes, BW 1544 Kbit, DLY 20000 usec,
> reliability 255/255, txload 1/255, rxload 1/255
> Encapsulation PPP, LCP Open, multilink Open, loopback not set
> Last input 00:00:00, output 00:00:01, output hang never
> Last clearing of "show interface" counters 05:13:27
> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> Queueing strategy: fifo
> Output queue: 0/40 (size/max)
> 2 minute input rate 0 bits/sec, 0 packets/sec
> 2 minute output rate 0 bits/sec, 0 packets/sec
> 179815 packets input, 246334673 bytes, 0 no buffer
> Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
> 1 input errors, 0 CRC, 1 frame, 0 overrun, 0 ignored, 0 abort
> 175062 packets output, 233229743 bytes, 0 underruns
> 0 output errors, 0 collisions, 2 interface resets
> 0 output buffer failures, 0 output buffers swapped out
> 0 carrier transitions
> DCD=up DSR=up DTR=up RTS=up CTS=up
>
> Serial0/1 is up, line protocol is up
> Hardware is PQUICC with Fractional T1 CSU/DSU
> Description: MLPPP T1 2
> MTU 1524 bytes, BW 1544 Kbit, DLY 20000 usec,
> reliability 255/255, txload 1/255, rxload 1/255
> Encapsulation PPP, LCP Open, multilink Open, loopback not set
> Last input 00:00:04, output 00:00:04, output hang never
> Last clearing of "show interface" counters 05:13:35
> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> Queueing strategy: fifo
> Output queue: 0/40 (size/max)
> 2 minute input rate 0 bits/sec, 0 packets/sec
> 2 minute output rate 0 bits/sec, 0 packets/sec
> 179809 packets input, 246361009 bytes, 0 no buffer
> Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
> 2 input errors, 1 CRC, 1 frame, 0 overrun, 0 ignored, 0 abort
> 175057 packets output, 233616175 bytes, 0 underruns
> 0 output errors, 0 collisions, 2 interface resets
> 0 output buffer failures, 0 output buffers swapped out
> 0 carrier transitions
> DCD=up DSR=up DTR=up RTS=up CTS=up
>
> Serial1/0 is up, line protocol is up
> Hardware is DSCC4 with integrated T1 CSU/DSU
> Description: MLPPP T1 3
> MTU 1524 bytes, BW 1544 Kbit, DLY 20000 usec,
> reliability 255/255, txload 1/255, rxload 1/255
> Encapsulation PPP, LCP Open, multilink Open, loopback not set
> Last input 00:00:05, output 00:00:05, output hang never
> Last clearing of "show interface" counters 05:13:40
> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> Queueing strategy: fifo
> Output queue: 0/40 (size/max)
> 2 minute input rate 0 bits/sec, 0 packets/sec
> 2 minute output rate 0 bits/sec, 0 packets/sec
> 179807 packets input, 246218180 bytes, 0 no buffer
> Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
> 5 input errors, 0 CRC, 5 frame, 0 overrun, 0 ignored, 0 abort
> 175054 packets output, 233403514 bytes, 0 underruns
> 0 output errors, 0 collisions, 2 interface resets
> 0 output buffer failures, 0 output buffers swapped out
> 0 carrier transitions
> DCD=up DSR=up DTR=up RTS=up CTS=up
>
>
>
>
>
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
