[c-nsp] How Cisco TAC troubleshoots router/switch crash using the stack frames?
Hello, 1) Am I correct that IOS, similarly to Linux kernel, has a separate stack for each process and each time a process makes a function call, a stack frame is added to stack containing information about function, it's arguments and variables and removed from stack once function returns to caller? As there is a command show stack PID I guess there is a stack per process.. 2) For example if following information is logged to crashinfo file during the L3 switch crash right before the forced reload: Traceback: 110DFFB8 10B04E98 11A391D0 116F6428 1172171C 10A8FA34 10A86BB4 Stack frames: Frame 1: pc=10B04E98 stack=20D0D670 Frame 2: pc=11A391D0 stack=20D0D678 Frame 3: pc=116F6428 stack=20D0D6B0 Frame 4: pc=1172171C stack=20D0D730 Frame 5: pc=10A8FA34 stack=20D0D7B0 Frame 6: pc=10A86BB4 stack=20D0D7B8 ..then how can TAC engineer trace back to certain function call? I understand that TAC engineers have probably access to IOS source code, symbol tables etc, but how how are the stack frames listed on Traceback line mapped to actual function calls and processes? regards, Martin ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] How Cisco TAC troubleshoots router/switch crash using the stack frames?
Hi, Not sure how knowing the method helps you at all, but the decoding is done using symbol tables which map the traceback values to function names which can be then searched for in the source code or similar bugs/cases in the database. Best regards, Andras On Mon, Jun 3, 2013 at 11:53 AM, Martin T m4rtn...@gmail.com wrote: Hello, 1) Am I correct that IOS, similarly to Linux kernel, has a separate stack for each process and each time a process makes a function call, a stack frame is added to stack containing information about function, it's arguments and variables and removed from stack once function returns to caller? As there is a command show stack PID I guess there is a stack per process.. 2) For example if following information is logged to crashinfo file during the L3 switch crash right before the forced reload: Traceback: 110DFFB8 10B04E98 11A391D0 116F6428 1172171C 10A8FA34 10A86BB4 Stack frames: Frame 1: pc=10B04E98 stack=20D0D670 Frame 2: pc=11A391D0 stack=20D0D678 Frame 3: pc=116F6428 stack=20D0D6B0 Frame 4: pc=1172171C stack=20D0D730 Frame 5: pc=10A8FA34 stack=20D0D7B0 Frame 6: pc=10A86BB4 stack=20D0D7B8 ..then how can TAC engineer trace back to certain function call? I understand that TAC engineers have probably access to IOS source code, symbol tables etc, but how how are the stack frames listed on Traceback line mapped to actual function calls and processes? regards, Martin ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] CRS-8-RP V09
Hello group, We had 4 faulty CRS-8-RP processors these last weeks. The same problem happened twice: CRS power-off, CRS power-on and the two RPs went faulty. There is a pattern. The SN begins with SAD132900xx and the Hw revision is 09: PID: CRS-8-RP , VID: V09, SN: SAD132900xx Anyone knows if there is a field notice for this ? We opened a case but TAC said there is no FN but I'm not convinced. More details about the failures. There are two cases: 1) the RP hangs just after the memory detection Initializing DDR SDRAM...found 4096 MB Initializing ECC on bank 0 Initializing ECC on bank 1 Initializing ECC on bank 2 Initializing ECC on bank 3 Turning off data cache, using DDR for first time [hangs] 2) the RP reboots when it is supposed to access the bootflash Initializing DDR SDRAM...found 4096 MB Initializing ECC on bank 0 Initializing ECC on bank 1 Initializing ECC on bank 2 Turning off data cache, using DDR for first time Initializing NVRAM... Testing a portion of DDR SDRAM ...done Reading ID EEPROMs ... ..Initializing SQUID ... Initializing PCI ... PCI0 device[1]: Vendor ID 0x10ee PCI0 device[1]: Device ID 0x300e PCI1 device[1]: Device ID 0x1100 PCI1 device[1]: Vendor ID 0x1013 PCI1 device[2]: Device ID 0x680 PCI1 device[2]: Vendor ID 0x1095 PCI1 device[3]: Device ID 0x5618 PCI1 device[3]: Vendor ID 0x14e4 Configuring MPPs ... Configuring PCMCIA slots ... System Bootstrap, Version 2.04(20110408:051659) [CRS ROMMON], Copyright (c) 1994-2011 by Cisco Systems, Inc. Acquiring backplane mastership ... successful Preparing for fan initialization. ready Setting fan speed to 4000 RPMs successful Reading backplane EEPROM ... Released backplane mastership ... Board type is 0x12 (1048578) Switch 0 initialized Enabling watchdog G4(7457-NonSMP-MV64360 Rev 4) platform with 4096 MB of main memory .. [reboot] We made some tests in the lab with one faulty card. This one hanged just after the Memory tests. After the RAM modules replacement, we got it like the others: the memory test was ok but the RP rebooted when it was supposed to load the image. This was a split boot setup. The disks are FAT32 formatted so the boot starts on the bootflash and the continues on the disks. I would say the RAM and/or the bootflash are the faulty components. Unfortunately the bootflash is not replaceable so we couldn't confirm this suspicion. The replacement RPs are V10 and V11. Thanks. Regards, Antonio Soares, CCIE #18473 (RS/SP) amsoa...@netcabo.pt http://www.ccie18473.net ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Fw: Re: IOS-XR BCP for EIGRP default route
replying to my own post- i meant ip default-network...sorry --- On Sun, 6/2/13, Randy randy_94...@yahoo.com wrote: From: Randy randy_94...@yahoo.com Subject: Re: [c-nsp] IOS-XR BCP for EIGRP default route To: cisco-nsp@puck.nether.net NSP cisco-nsp@puck.nether.net, Jason Lixfeld ja...@lixfeld.ca Date: Sunday, June 2, 2013, 8:43 PM IIRC default-info originate for eigrp was introduced in nx-os. ios and ios-xr don't have it. ios for eigrp had this idiotic concept carried-over from IGRP of default-gateway. end rant ./Randy --- On Sun, 6/2/13, Jason Lixfeld ja...@lixfeld.ca wrote: From: Jason Lixfeld ja...@lixfeld.ca Subject: [c-nsp] IOS-XR BCP for EIGRP default route To: cisco-nsp@puck.nether.net NSP cisco-nsp@puck.nether.net Date: Sunday, June 2, 2013, 12:26 PM Hi, In reading the documentation for EIGRP default route advertisement into EIGRP, I'm finding only a few ways to accomplish this, based on my current topology. In vrf foo, I have a default route redistributed from static on an upstream PE router. I want to get that default route sent towards a CE router downstream of my EIGRP-PE. Near as I can tell, I can do one of the following to accomplish that: 1. Redistribute BGP into EIGRP on EIGRP-PE 2. Create a summary address on the CE facing interface on EIGRP-PE 3. Create a static default on EIGRP-PE and redistribute static into EIGRP. Option 1 seems a little heavy handed relative to the very simple method of default-information originate in a protocol like ISIS. Option 2 seems a little administratively prohibitive in the sense of having to add summary-address 100 0.0.0.0 0.0.0.0 250 on each and every CE interface. Option 3 would require lots of peripheral configuration to ensure that default didn't somehow make it's way back upstream. Is there something a little more elegant or am I faced with one of these three options? In a perfect world, isis-like default-information originate would be loverly, but I'm not sure if that's possible in EIGRP land. Thanks in advance. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] CRS-8-RP V09
I forgot to mention that in the lab, and after replacing the faulty RAM modules, we were able to go to Rommon. In Rommon, the dir bootflash: caused a reboot and this was systematic. Thanks. Regards, Antonio Soares, CCIE #18473 (RS/SP) amsoa...@netcabo.pt http://www.ccie18473.net -Original Message- From: cisco-nsp [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Antonio Soares Sent: segunda-feira, 3 de Junho de 2013 13:37 To: cisco-nsp@puck.nether.net Subject: [c-nsp] CRS-8-RP V09 Hello group, We had 4 faulty CRS-8-RP processors these last weeks. The same problem happened twice: CRS power-off, CRS power-on and the two RPs went faulty. There is a pattern. The SN begins with SAD132900xx and the Hw revision is 09: PID: CRS-8-RP , VID: V09, SN: SAD132900xx Anyone knows if there is a field notice for this ? We opened a case but TAC said there is no FN but I'm not convinced. More details about the failures. There are two cases: 1) the RP hangs just after the memory detection Initializing DDR SDRAM...found 4096 MB Initializing ECC on bank 0 Initializing ECC on bank 1 Initializing ECC on bank 2 Initializing ECC on bank 3 Turning off data cache, using DDR for first time [hangs] 2) the RP reboots when it is supposed to access the bootflash Initializing DDR SDRAM...found 4096 MB Initializing ECC on bank 0 Initializing ECC on bank 1 Initializing ECC on bank 2 Turning off data cache, using DDR for first time Initializing NVRAM... Testing a portion of DDR SDRAM ...done Reading ID EEPROMs ... ..Initializing SQUID ... Initializing PCI ... PCI0 device[1]: Vendor ID 0x10ee PCI0 device[1]: Device ID 0x300e PCI1 device[1]: Device ID 0x1100 PCI1 device[1]: Vendor ID 0x1013 PCI1 device[2]: Device ID 0x680 PCI1 device[2]: Vendor ID 0x1095 PCI1 device[3]: Device ID 0x5618 PCI1 device[3]: Vendor ID 0x14e4 Configuring MPPs ... Configuring PCMCIA slots ... System Bootstrap, Version 2.04(20110408:051659) [CRS ROMMON], Copyright (c) 1994-2011 by Cisco Systems, Inc. Acquiring backplane mastership ... successful Preparing for fan initialization. ready Setting fan speed to 4000 RPMs successful Reading backplane EEPROM ... Released backplane mastership ... Board type is 0x12 (1048578) Switch 0 initialized Enabling watchdog G4(7457-NonSMP-MV64360 Rev 4) platform with 4096 MB of main memory .. [reboot] We made some tests in the lab with one faulty card. This one hanged just after the Memory tests. After the RAM modules replacement, we got it like the others: the memory test was ok but the RP rebooted when it was supposed to load the image. This was a split boot setup. The disks are FAT32 formatted so the boot starts on the bootflash and the continues on the disks. I would say the RAM and/or the bootflash are the faulty components. Unfortunately the bootflash is not replaceable so we couldn't confirm this suspicion. The replacement RPs are V10 and V11. Thanks. Regards, Antonio Soares, CCIE #18473 (RS/SP) amsoa...@netcabo.pt http://www.ccie18473.net ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] How Cisco TAC troubleshoots router/switch crash using the stack frames?
On Mon, Jun 3, 2013 at 4:53 AM, Martin T m4rtn...@gmail.com wrote: Hello, 1) Am I correct that IOS, similarly to Linux kernel, has a separate stack for each process and each time a process makes a function call, a stack frame is added to stack containing information about function, it's arguments and variables and removed from stack once function returns to caller? As there is a command show stack PID I guess there is a stack per process.. Sort of. Except that everything is known in (classic) IOS - all code and all addresses are known, so when the below happens, Cisco knows exactly what function call was executed since each process shares all the code address space. IIRC some IOS really are actually one big monolithic process, what it calls processes aren't really fully independent processes but more like a task queue or similar. And since IOS doesn't dynamically load code, addresses (pc) are fully known for a given version. They can map back to source code lines very easily. 2) For example if following information is logged to crashinfo file during the L3 switch crash right before the forced reload: Traceback: 110DFFB8 10B04E98 11A391D0 116F6428 1172171C 10A8FA34 10A86BB4 Stack frames: Frame 1: pc=10B04E98 stack=20D0D670 Frame 2: pc=11A391D0 stack=20D0D678 Frame 3: pc=116F6428 stack=20D0D6B0 Frame 4: pc=1172171C stack=20D0D730 Frame 5: pc=10A8FA34 stack=20D0D7B0 Frame 6: pc=10A86BB4 stack=20D0D7B8 ..then how can TAC engineer trace back to certain function call? I understand that TAC engineers have probably access to IOS source code, symbol tables etc, but how how are the stack frames listed on Traceback line mapped to actual function calls and processes? regards, Martin ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ -- Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds. -- Samuel Butler ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Combine two 2-wire DSL
Hi, is it possible to combine two 2-wire DSL's into one 4-wire DSL using Cisco 4-wire DSL card on 1841? ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Equivalent of ip multicast boundary on N7k for blocking data packets?
On our 6500s, we have a standard multicast config inspired by the Cymru secure IOS template: http://www.cymru.com/Documents/secure-ios-template.html ...in our case, looking broadly like this: ip access-list standard MULTICAST-in deny 224.0.1.60 deny 224.77.0.0 0.0.255.255 deny 226.77.0.0 0.0.255.255 permit 239.192.0.0 0.3.255.255 deny 239.0.0.0 0.255.255.255 permit 224.0.0.0 15.255.255.255 int VlanXX description subnet facing some clients ... ip pim sparse-mode ip multicast boundary MULTICAST-in ip igmp access-group MULTICAST-in The two multicast/ACL commands accomplish two different things - the boundary prevents clients sending to that group, the igmp join prevents clients asking to receive from that group. This lets us block some common junk - such as Symantec Ghost (sessions trample on each other) and groups from 239/8 but not 239.192/14 - that in the past have caused masses of multicast routing entries to no real purpose. How can I accomplish the equivalent of the boundary on NX-OS 5.2 for N7k, given it lacks the command? Does one just use a normal ACL, and if so, are there any caveats to doing so e.g. does boundary do *other* things that a plain ACL would miss? It's important to note that the boundary translates into TCAM entries on 6500s i.e. it really does stop traffic forwarding, and this has proven important to us in the past: #sh tcam interface vlxx acl in ip * Global Defaults not shared Entries from Bank 0 permit ip any any Entries from Bank 1 permit ip any 239.192.0.0 0.3.255.255 deny ip any 224.77.0.0 0.0.255.255 deny ip any 226.77.0.0 0.0.255.255 deny ip any host 224.0.1.60 deny ip any 239.0.0.0 0.255.255.255 permit ip any any ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Equivalent of ip multicast boundary on N7k for blocking data packets?
At 01:08 PM 6/3/2013 Monday, Phil Mayers clamored: How can I accomplish the equivalent of the boundary on NX-OS 5.2 for N7k, given it lacks the command? Does one just use a normal ACL, and if so, are there any caveats to doing so e.g. does boundary do *other* things that a plain ACL would miss? In n7k, you must use a combination of control plane data plane filtering to get the equivalent functionality of multicast boundary. For data plane, it's nothing more than ip access-group with matches on multicast traffic. For control plane, there is independent filtering control for each protocol, ie, PIM, IGMP, MSDP, etc. Hope that helps, Tim Tim Stevenson, tstev...@cisco.com Routing Switching CCIE #5561 Distinguished Technical Marketing Engineer, Cisco Nexus 7000 Cisco - http://www.cisco.com IP Phone: 408-526-6759 The contents of this message may be *Cisco Confidential* and are intended for the specified recipients only. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Equivalent of ip multicast boundary on N7k for blocking data packets?
On 03/06/2013 21:44, Tim Stevenson wrote: At 01:08 PM 6/3/2013 Monday, Phil Mayers clamored: How can I accomplish the equivalent of the boundary on NX-OS 5.2 for N7k, given it lacks the command? Does one just use a normal ACL, and if so, are there any caveats to doing so e.g. does boundary do *other* things that a plain ACL would miss? In n7k, you must use a combination of control plane data plane filtering to get the equivalent functionality of multicast boundary. For data plane, it's nothing more than ip access-group with matches on multicast traffic. Ok, so just use ACLs. Out of curiosity, what was the rationale for merging the unicast and multicast data-plane filtering? The IOS-style split seems to have some advantages in terms of manageability, including that you don't need to write both ingress and egress ACLs. Though I suppose the latter are more flexible. For control plane, there is independent filtering control for each protocol, ie, PIM, IGMP, MSDP, etc. Indeed. It's somewhat non-obvious to me at this time how the PIM control plane filtering interacts with data- and IGMP-driven PIM events on edge subnets, but I probably need to re-read the docs. Hope that helps, Very much so, thanks. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] ASR9k HSRP group reuse
Hi Aivars, The bottom line is - you can not re-use them on the same physical interface unless you can be sure that all groups with the same id will switch-over at the same time (I'm not sure if the limitation is on the interface level or NP level). VRRP v3 helps here, as you can have more ids, but i'm not sure if it's implemented in 9k already. kind regards Pshem On 3 June 2013 22:26, Aivars aiv...@ml.lv wrote: Hi, Recently I encountered HSRP problem on ASR9k and as far as I understand it is related to HSRP group reuse. My previous HSRP knowledge stated that group number only determines stanby IP MAC address and it can be resued as log as it is unique inside a broadcast domain. During the incident investigation I found Alexander Thuijs comment https://supportforums.cisco.com/thread/2216914 about group reuse. I would like to understand the problem in detail. As far as I got it, when we use bridge domains, each NP has a MAC table. As soon as one of the groups fails over, NP gets a request do remove it and it is entirely removed even for another bridge domain. Did I get it right? Could someone please clarify HSRP group reuse un 9k? It would be nice to have this covered in configuration guide. Aivars ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Combine two 2-wire DSL
It requires the other end (exchange end) to be plugged into an equivalent type card on the dslam that can combine the 4-wires into a single service. Can you even do ADSL on 4-wire ? I know SHDSL can be done over 4-wire. regards, Tony. From: Pavel Dimow paveldi...@gmail.com To: cisco-nsp@puck.nether.net cisco-nsp@puck.nether.net Sent: Tuesday, 4 June 2013 5:46 AM Subject: [c-nsp] Combine two 2-wire DSL Hi, is it possible to combine two 2-wire DSL's into one 4-wire DSL using Cisco 4-wire DSL card on 1841? ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
