I just did this lab yesterday and didn't have any built-in issues with the frame relay links. If I were in that situation I would probably revert the rack, not load any configs and see if I could get the frame links working that way. If you can't get them working that way and you're confident your base frame config should work it could certainly be a hardware issue.
On Sat, Feb 26, 2011 at 2:37 PM, Don Pezet <[email protected]> wrote: > Good afternoon everyone, > > I had an interesting issue creep up today while I was studying Multicast > routing using ProctorLabs this morning. I was working on IPExpert Lab > Preparation Workbook Vol. 1 Lab 25 for the v4.0 blueprint (the workbook is > actually version 11.0). I skipped my normal pattern and didn't verify > network connectivity before starting because I had done this lab before and > I knew there were no sneaky layer 2 or layer 3 problems hidden in waiting. > I > was almost through the lab when I noticed that my frame-relay interfaces > between R2, R4 and R6 were not up. I was on Task 25.5b when I noticed that > MSDP wasn't peering up between R2 and R6. I did a quick examination and > noticed that the serial interfaces themselves were down. I went ahead and > finished the lab (I only had one more task left) and then set about > troubleshooting the frame-relay connection. I am pretty comfortable with > frame-relay, but what I found stumped me. After troubleshooting a bit, I > determined that it looked like a hardware issue. I was a little nervous > about this, as every time I have heard of someone saying they found a > hardware issue when it comes to CCIE labs they have turned out wrong. So, I > backed up my configs, reverted my entire pod and reloaded the Vol. 1 Lab 25 > initial configs over again. Well, the problem was still there. At this > point, I opened up a ticket with after hours support, and they assured me > it > was a configuration issue. I did some more troubleshooting and with a > little > trepidation I went back to them again. They said they would escalate the > problem to Tier 2 on Monday, but for now, I was stuck. They suggested I > post > my problem to the OSL and see what you guys think. Any help on this is much > appreciated. I'll post the data from one of my routers and see what we can > find. Here is the troubleshooting information I gathered for R2, although > the problem existed on R4 and R6 identically. > > On R2, my Serial0/1/0 interface was configured as follows (straight from > the > initial configs): > > interface Serial0/1/0 > no ip address > encapsulation frame-relay > no arp frame-relay > no frame-relay inverse-arp > frame-relay lmi-type cisco > ! > interface Serial0/1/0.24 point-to-point > ip address 150.50.24.2 255.255.255.252 > frame-relay interface-dlci 204 > ! > interface Serial0/1/0.26 point-to-point > ip address 150.50.26.2 255.255.255.252 > frame-relay interface-dlci 206 > ! > > If you would like the whole config, you can look at either the initial or > final configs from Lab 25 as I had the same symptoms under both. > > Here's the output from "show interface s0/1/0" > > Serial0/1/0 is down, line protocol is down > Hardware is GT96K Serial > MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec, > reliability 255/255, txload 1/255, rxload 1/255 > Encapsulation FRAME-RELAY, loopback not set > Keepalive set (10 sec) > CRC checking enabled > LMI enq sent 0, LMI stat recvd 0, LMI upd recvd 0, DTE LMI down > LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0 > LMI DLCI 1023 LMI type is CISCO frame relay DTE > FR SVC disabled, LAPF state down > Broadcast queue 0/64, broadcasts sent/dropped 0/0, interface broadcasts 0 > Last input never, output never, output hang never > Last clearing of "show interface" counters 01:16:33 > Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 > Queueing strategy: weighted fair > Output queue: 0/1000/64/0 (size/max total/threshold/drops) > Conversations 0/0/256 (active/max active/max total) > Reserved Conversations 0/0 (allocated/max allocated) > Available Bandwidth 1158 kilobits/sec > 5 minute input rate 0 bits/sec, 0 packets/sec > 5 minute output rate 0 bits/sec, 0 packets/sec > 0 packets input, 0 bytes, 0 no buffer > Received 0 broadcasts, 0 runts, 0 giants, 0 throttles > 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort > 0 packets output, 0 bytes, 0 underruns > 0 output errors, 0 collisions, 4 interface resets > 0 unknown protocol drops > 0 output buffer failures, 0 output buffers swapped out > 0 carrier transitions > DCD=down DSR=down DTR=up RTS=up CTS=down > > You'll notice that the line is down/down. Now, even if my config was > completely wrong and I had done something silly like configured the > interface for PPP it should still show as up/down right? This was what got > me thinking that this was a hardware issue. You'll also notice "DTE LMI > down" which isn't a good thing. This meant that I wasn't even communicating > with the frame-relay switch. So, I fired up debug and took a look at what > was going on. I turned on debugging for frame-relay packet, frame-relay lmi > and frame-relay detailed. I think shutdown the s0/1/0 interface and brought > it back up again. Here is what I got: > > Feb 26 17:30:45.959: Serial0/1/0.24: encaps failed--line down > Feb 26 17:30:45.959: Serial0/1/0.26: encaps failed--line down > > That's it. Just those two lines. Well, you can't see a frame-relay switch > over an interface that is down so I double checked R4 and R6 and they were > identical. The odds of having all three routers with a serial port failure > was too unlikely, so I started looking to the frame-relay switch side of > things. The next thing I looked at was "show frame-relay lmi" whose output > follows: > > LMI Statistics for interface Serial0/1/0 (Frame Relay DTE) LMI TYPE = CISCO > Invalid Unnumbered info 0 Invalid Prot Disc 0 > Invalid dummy Call Ref 0 Invalid Msg Type 0 > Invalid Status Message 0 Invalid Lock Shift 0 > Invalid Information ID 0 Invalid Report IE Len 0 > Invalid Report Request 0 Invalid Keep IE Len 0 > Num Status Enq. Sent 0 Num Status msgs Rcvd 0 > Num Update Status Rcvd 0 Num Status Timeouts 0 > Last Full Status Req never Last Full Status Rcvd never > > That definitely didn't look good. 0 messages of any sort had been sent, > good > or bad. This is what I would expect to see on a router that had absolutely > nothing plugged into the serial port. So, that's what I checked next with > "show controller s0/1/0" > > Interface Serial0/1/0 > Hardware is GT96K > DTE V.35 clocks stopped. > idb at 0x686DC3CC, driver data structure at 0x686DD908 > wic_info 0x686DDF2C > Physical Port 3, SCC Num 3 > <output omitted> > > I cut this one short for the e-mail because the relevant part is right up > top: DTE V.35 clocks stopped. That's when I decided this had to be an error > with the frame-relay switch. I could see that a cable was present, but I > wasn't receiving any clock signaling from the switch. I used the web > interface for ProctorLabs to power cycle the frame-relay switch, but that > didn't make any difference. I couldn't even tell whether or not it had > rebooted since my routers couldn't even detect the frame-relay switch at > layer 1. That's all I could do since we don't have access to the > frame-relay > switch for troubleshooting. So I double checked everything again, and then > opened up an after hours support ticket, which is what led me to here. > > I think I've done my due diligence on this one, but do you guys see > anything > I missed? Can you think of something that might cause this? If I had to > guess, I would say that the frame-relay switch was actually powered off, > however support assures me it is on and operational. Any thoughts, input, > ideas, or criticism on my troubleshooting are appreciated. Problems like > these are simply another opportunity to learn. > > Thanks everyone, > > Don Pezet > _______________________________________________ > For more information regarding industry leading CCIE Lab training, please > visit www.ipexpert.com > _______________________________________________ For more information regarding industry leading CCIE Lab training, please visit www.ipexpert.com
