I was trouble shooing my network today and found a nasty little bug
when some
one does 'show isis database' from exec mode on C181X Software
(C181X-ADVIPSERVICESK9-M), Version 15.0(1)M1, IOS.
After issuing the command you get the output of it, and some time in
the next
30 sec the router crashes.
example:
LTRKAKHQR01-c1811w#sh isis database
IS-IS Level-1 Link State Database:
LSPID LSP Seq Num LSP Checksum LSP Holdtime
ATT/P/OL
STPBFLGURT1.00-00 0x00005F0A 0x49A3 907
0/0/0
galaxydoor.00-00 0x0000200A 0x2DF0 900
0/0/0
LTRKAKHQR01-c1.00-00* 0x00000953 0x64C1 1099
0/0/0
TAMQFLTART1.00-00 0x00005859 0x1542 908
0/0/0
IS-IS Level-2 Link State Database:
LSPID LSP Seq Num LSP Checksum LSP Holdtime
ATT/P/OL
STPBFLGURT1.00-00 0x000060A8 0x1149 914
0/0/0
galaxydoor.00-00 0x0000200F 0x645F 912
0/0/0
LTRKAKHQR01-c1.00-00* 0x00000991 0xF41F 916
0/0/0
TAMQFLTART1.00-00 0x00005926 0x83FD 913
0/0/0
LTRKAKHQR01-c1811w#term mon
LTRKAKHQR01-c1811w#sh clock
01:44:47.438 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:44:56.418 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:01.690 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:06.182 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:10.146 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:12.658 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:16.222 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
______BAM! Lockup at this point______
from the log output:
Feb 6 20:45:27 192.168.3.210 103: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:20
UTC: %SYS-3-CPUHOG: Task is running for (2000)msecs, more than (2000)
msecs
(0/0),process = Check heaps.
Feb 6 20:45:27 192.168.3.210 104: LTRKAKHQR01-c1811w: -Traceback=
0x8007CCB0z
0x80B20C18z 0x80B22C8Cz 0x80B20EC8z 0x82050E18z 0x82052364z
0x82052770z
0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z
Feb 6 20:45:27 192.168.3.210 105: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:22
UTC: %SYS-3-CPUHOG: Task is running for (4000)msecs, more than (2000)
msecs
(0/0),process = Check heaps.
Feb 6 20:45:27 192.168.3.210 106: LTRKAKHQR01-c1811w: -Traceback=
0x8007CCB8z
0x80B20C18z 0x80B22528z 0x80B20EC8z 0x820500E4z 0x82050E54z
0x82052364z
0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z
Feb 6 20:45:27 192.168.3.210 107: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-3-BADMAGIC: Corrupt block at 86AC28DC (magic 813E0508), -
Traceback=
0x82052388z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz
0x80124418z
Feb 6 20:45:27 192.168.3.210 108: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-6-MTRACE: mallocfree: addr, pc
Feb 6 20:45:27 192.168.3.210 109: LTRKAKHQR01-c1811w:
86297A44,80BAE58C
86297A44,40000294 86CC08A0,80BAE570 86CC08A0,3000021E
Feb 6 20:45:27 192.168.3.210 110: LTRKAKHQR01-c1811w:
86DC8180,8154FFC0
866536C8,8154FE24 866536C8,8154FE24 866536C8,8154FE88
Feb 6 20:45:27 192.168.3.210 111: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-6-MTRACE: mallocfree: addr, pc
Feb 6 20:45:27 192.168.3.210 112: LTRKAKHQR01-c1811w:
8666F860,81569A98
866536C8,8154FE88 866536C8,8154FE24 866536C8,8154FE24
Feb 6 20:45:27 192.168.3.210 113: LTRKAKHQR01-c1811w:
866536C8,8154EE70
866536C8,8154EE70 866536C8,8154EE70 866536C8,8154EE70
Feb 6 20:45:27 192.168.3.210 114: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-6-BLKINFO: Corrupted magic value in in-use block blk
86AC28DC, words
6002, alloc 8012DAC4, InUse, dealloc FFFFFFFF, rfcnt 1, -Traceback=
0x82010150z 0x82052618z 0x82052770z 0x82055410z 0x820555CCz
0x8012086Cz
0x80124418z
Feb 6 20:45:28 192.168.3.210 115: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-6-MEMDUMP: 0x86AC28DC: 0x813E0508 0x0 0x0 0x8364CCBC
Feb 6 20:45:28 192.168.3.210 116: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-6-MEMDUMP: 0x86AC28EC: 0x8012DAC4 0x86AC57F0 0x86AB9C1C 0x80001772
Feb 6 20:45:28 192.168.3.210 117: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-6-MEMDUMP: 0x86AC28FC: 0x1 0x86AC2980 0x15C 0x86D41800
Feb 6 20:45:28 192.168.3.210 118: LTRKAKHQR01-c1811w: Feb 7 2010
01:45:24
UTC: %SYS-6-STACKLOW: Stack for process Virtual Exec running low,
12/12000
Feb 6 20:45:40 192.168.3.250 1038: TAMQFLTART1: Feb 7 2010
01:45:39 UTC:
%LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed
state to down
-------
Some times I'll get a crashinfo file, other times I will not.
From a previous crash info:
-------------
CMD: 'sh isis database' 21:23:27 UTC Sat Feb 6 2010
validblock_diagnose, code = 2
current memory block, bp = 0x8700E0B0,
memorypool type is Processor
data check, ptr = 0x8700E0E0
next memory block, bp = 0x87010FC4,
memorypool type is Processor
data check, ptr = 0x87010FF4
previous memory block, bp = 0x870053DC,
memorypool type is Processor
data check, ptr = 0x8700540C
========= Dump bp = 0x8700E0B0 ======================
8700DFB0: 0 8700EAB0 FFFFFFFF 0 0
0 0 0
8700DFD0: 0 0 6347E519 0 8207070C D02688F2
6347E519 85F7C994
8700DFF0: 85F7C994 811AEC4C 8700E010 811AB16C D0D0D0D 245EBB78
D0D0D0D 867CD2F4
8700E010: 8700E040 813B2838 D0D0D0D D0D0D0D D0D0D0D D0D0D0D
D0D0D0D 82070710
8700E030: 85F7C994 6347E519 875E3258 875E3214 8700E070 813B2B8C
D0D0D0D 48822022
8700E050: 1 8700E2D8 0 0 8700E070 6347E519
8700E400 0
8700E070: 8700E0B0 813B4470 0 0 28822022
6347E519 0 0
8700E090: 0 0 6347E519 85F7C994 0 0
8700E400 0
8700E0B0: 8700E350 813E0508 0 0 8012DAC4 87010FC4
870053F0 80001772
8700E0D0: 1 0 8700E158 872550DC FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF
8700E0F0: 0 0 FFFFFFFF FFFFFFFF FFFF FFFFFFFF
FFFFFFFF FFFFFFFF
8700E110: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF 0
8700E130: 2F 86427028 0 85F7C994 0 6347E519
6347E519 245EBB78
8700E150: 8700E2F0 867CD2F4 8700E1D8 811ABDE0 FFFFFFFF 6347E519
D02688F2 2F
8700E170: 0 0 0 C0 FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF
8700E190: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF
=========
Feb 6 2010 21:24:09 UTC: %SYS-3-CPUHOG: Task is running for (2000)
msecs, more
than (2000)msecs (1/1),process = Check heaps.
-Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22C8Cz 0x80B20EC8z
0x82050E18z
0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz
0x80124418z Dump
bp->next = 0x87010FC4 ======================
87010EC4: 61780000 87010EF0 84228082 73796E74 A4CB80 7002FD0
87010F20 87010E70
87010EE4: 87010EF8 83EB0000 83EB0000 0 83EB0000 0
A4CB80 0
87010F04: 0 867F2054 0 0 86493648 87010FB0
80B77310 FFFFFF
87010F24: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF
87010F44: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
2A 1 FFFFFFFF
87010F64: FFFFFFFF 0 0 0 0
0 0 0
87010F84: 0 0 0 0 0
0 0 0
87010FA4: 0 0 0 87010FB8 8012086C 0 80124418
FD0110DF
87010FC4: AB1234CD E40000 15F 873074D0 80B4FCA0 87015E18
8700E0C4 80002712
87010FE4: 1 8200EA4C 166 872550DC 0 0 87307494
0
87011004: 87307494 258 2C7 140018 2C1
0 0 0
87011024: 0 430000 83EC2BBC 41414120 536D616C 6C204368
756E0000 87011B6C
87011044: 87015E14 0 0 87011B70 87011B88 87011BA0
87011BB8 87011BD0
87011064: 87011BE8 87011C00 87011C18 87011C30 87011C48 87011C60
87011C78 87011C90
87011084: 87011CA8 87011CC0 87011CD8 87011CF0 87011D08 87011D20
87011D38 87011D50
870110A4: 87011D68 87011D80 87011D98 87011DB0 87011DC8 87011DE0
87011DF8 87011E10
========== Dump bp->previous = 0x870053F0 =====================
870052F0: 0 0 0 0 0
0 0 0
87005310: 0 FD0110DF AB1234CD FFFE0000 0 82FC74AC
81BDE144 87005390
87005330: 870052A8 80000024 1 0 1 850B5B2C
83824BA0 0
87005350: 0 0 1 0
Feb 6 2010 21:24:11 UTC: %SYS-3-CPUHOG: Task is running for (4000)
msecs, more
than (2000)msecs (1/1),process = Check heaps.
-Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22528z 0x80B20EC8z
0x820500E4z
0x82050E54z 0x82052364z 0x82052770z 0x82055410z 0x820555CCz
0x8012086Cz
0x80124418z 0 0 0 90000
87005370: 1 870051FC 0 0 0
0 0 FD0110DF
87005390: AB1234CD FFFE0000 0 82FC74AC 81BD9178 870053DC
8700532C 8000000E
870053B0: 1 0 1 850B5B2C 1
0 0 0
870053D0: 0 0 FD0110DF AB1234CD 750000 75
83646E94 82C4EED4
870053F0: 8700E0B0 870053A4 4652 0 82C89068 7C
850B1410 DEADBEEF
87005410: 82C89068 0 D0D0D0D 83EC321C 83EC3218 D0D0D0D
D0D0D0D D0D0D0D
87005430: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
D0D0D0D D0D0D0D
87005450: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
D0D0D0D D0D0D0D
87005470: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
D0D0D0D D0D0D0D
87005490: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
D0D0D0D D0D0D0D
870054B0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
D0D0D0D D0D0D0D
870054D0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
D0D0D0D D0D0D0D
============================================
Feb 6 2010 21:24:12 UTC: %SYS-3-BADMAGIC: Corrupt block at 8700E0B0
(magic
8700E350), -Traceback= 0x82052388z 0x82052770z 0x82055410z
0x820555CCz
0x8012086Cz 0x80124418z
Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
873068BC,80BAE58C 873068BC,40000294 859EC9B4,80BAE570
859EC9B4,3000021E
86EAC770,8154FFC0 86EA8998,8154FE24 86EA8998,8154FE24
859EC9B4,81540E3C
Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
859EC9B4,8153B354 859EC9B4,3000021E 86EA8998,8154FE88
86EAB098,81569A98
86EA8998,8154FE88 86EA8998,8154FE24 86EA8998,8154FE24
86EA8998,8154EE70
Feb 6 2010 21:24:12 UTC: %SYS-6-BLKINFO: Corrupted magic value in
in-use
block blk 8700E0B0, words 6002, alloc 8012DAC4, InUse, dealloc
FFFFFFFF, rfcnt
1, -Traceback= 0x82010150z 0x82052618z 0x82052770z 0x82055410z
0x820555CCz
0x8012086Cz 0x80124418z
Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0B0: 0x8700E350
0x813E0508
0x0 0x0
Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0C0: 0x8012DAC4
0x87010FC4
0x870053F0 0x80001772
Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0D0: 0x1 0x0
0x8700E158
0x872550DC
%Software-forced reload
21:24:12 UTC Sat Feb 6 2010: Unexpected exception to CPU: vector
1500, PC =
0x8011E220, LR = 0x8011E1E4
-Traceback= 0x8011E220z 0x8011E1E4z 0x82052770z 0x82055410z
0x820555CCz
0x8012086Cz 0x80124418z
CPU Register Context:
MSR = 0x02029220 CR = 0x28000042 CTR = 0x81F26400 XER = 0x00000000
R0 = 0x8011E1E4 R1 = 0x8511CBA8 R2 = 0xFFE97C10 R3 =
0x83FA9978
R4 = 0x82F869BC R5 = 0x00000000 R6 = 0x83970000 R7 =
0x82F60000
R8 = 0x02029220 R9 = 0x83AD0000 R10 = 0x00000000 R11 = 0x00000000
R12 = 0x24000088 R13 = 0xFFE994A8 R14 = 0x820554DC R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x83D60000 R23 =
0x83D60000
R24 = 0xAB1234AB R25 = 0xAB1234CD R26 = 0x83D60000 R27 =
0x85FBD91C
R28 = 0x00000000 R29 = 0x83647534 R30 = 0x83980000 R31 = 0x00000000
------
I've tried this on both 1811w's on my network and had the exact same
problems.
Any one else seen this or know if it's a known bug? I've searched
the cisco
site and cannot find a reference to this issue.
--
Bryan Fields
727-409-1194 - Voice
727-214-2508 - Fax
http://bryanfields.net
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
