Hi all, We also have this problem for about a year now but exclusively on 3600’s, although 2600 and 3700 are not beyond suspicion, our 702, 1140, 1810, 2700’s seem to be fine. It also looked like we were the only ones with this problem but there are more. So get on to your supplier and Cisco that this is a serious issue and needs fixing.
I seems to be at least in all 8.2 and 8.3 releases. We have TAC-case SR 682811103 running for this and we are currently running a 8.2.166.0 based debug version testing out a possible fix. What seems to be the case is that the flash file system gets corrupted. Not surprisingly when the AP needs to reboot it runs into all kind of problems, like a not working boot image, not loading radio firmware or corrupt config. The AP drops to boot rom or gets in to a boot loop. The only remedy is via the console do fsck or format of the flash and to reload either the current image or the recovery image from a tftp server. The problem is not easy to debug as there are no indications of a running AP which is corrupt and the trigger is as yet unknown, it is however detectable remotely. We have developed a script which checks the AP’s and with some hidden features re-installs the image if it is corrupted. Of our 400+ AP3600’s there are about 10 fails a week, leave the check longer and the numbers go up. This script catches most corrupt AP’s before they break on a reboot, it is highly tailored so it won’t easily translate to a different environment and of course it is not a fix. Regards, Jan Freerk Popma | ICT Service Center, Networkmanagement | University of Twente, Enschede, Netherlands Building Citadel room 219 | T: +31 53 489 4321 | j.f.po...@utwente.nl<mailto:j.f.po...@utwente.nl> | www.utwente.nl<http://www.utwente.nl> From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Tristan Gulyas Sent: dinsdag 12 december 2017 07:18 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Cisco AP 'flash' bug Hi all, I was under the impression that we were the only customer who have been hitting this. 8.3.112.7 engineering release. We've seen it on all platforms - fixed in 702W in our current release (we believe) but we're seeing it on 1532, 3502, 3602, 2702, 3702. Not present on 3800/1562 from what we've seen. One catalyst for this has been AP reboots. Has anyone else been hit by this bug or been provided with a fix? Cheers, Tristan -- TRISTAN GULYAS Senior Network Engineer Technology Services, eSolutions Monash University 738 Blackburn Road Clayton 3168 Australia T: +61 3 9902 9092 E: tristan.gul...@monash.edu<mailto:tristan.gul...@monash.edu> monash.edu<http://monash.edu/> On 20 Jan 2017, at 7:46 am, McClintic, Thomas <thomas.mcclin...@uth.tmc.edu<mailto:thomas.mcclin...@uth.tmc.edu>> wrote: Next time you have this issue, try connecting a console to the AP and run the following: ap: fsck flash: Are you sure you want to fsck "flash:" (could take some time) (y/n)?y flashfs[0]: …………… ap: boot This works for us on the failed to reload properly APs. From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Garret Peirce Sent: Thursday, January 19, 2017 10:44 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Cisco AP 'flash' bug Ian, thanks for the response. To commiserate it does feel that wireless ecosystem has been affected by a larger bloom of bugs over the last year or so. Some of that may be due to enhanced vigilance and our tracking them down to root causes, but whatever the case, in aggregate it's a concern here as well. Another related statistic about this issue. With ~7000 total APs potentially affected we're seeing an incidence rate below 1% which although low, it's felt more when you're making fire-fighting trips to visit/replace affected APs. On Thu, Jan 19, 2017 at 10:28 AM, Ian Lyons <ily...@rollins.edu<mailto:ily...@rollins.edu>> wrote: Yes, we own that bug too. Pretty much we have every bug ..and have been patching like madmen since July. From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>] On Behalf Of Garret Peirce Sent: Thursday, January 19, 2017 10:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Cisco AP 'flash' bug Over the last few months we've run into/discovered a Cisco bug and I was curious if any in this community have been seeing it as well. In a nutshell, it appears the flash is being corrupted and the AP then enters a boot loop or fails to boot at all. We are apparently seeing a failure rate of roughly 10 APs per month. My engineer's summary is below. ================= CSCvc74528 description is below, but it fails to take into account that occasionally the boot loop doesn't happen and the AP will just crash on boot, or fail to boot at all. Working with them to add some things to the description. "APs go into boot cycle due to corrupt image, do not download new image from WLC CSCvc74528 Description Symptom: APs reboot and when booting back up the image gets corrupted. The AP checks the WLC and sees it has the same image in flash and does not download the WLC image. The image on the AP is corrupt and therefor continuously reboots into the corrupted image. Conditions: 2702I, 3602I and 3702I APs on a 8540 WLC running 8.2.141.0 or 8.3.102.0 code do not download WLC code due to same image on flash. Bad flash in APs Workaround: Format APs via console with new image, holds for a few reboots. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.educause.edu_discuss&d=DQMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=rYfqH_8oTvcXxRxUI3x3m3Y7Nwgir7tnuoGbdZsrUM4&m=hjvPaJDEwbeTBYMagZWhbrzxuF4zzIipa26zlRB9_9c&s=AKNZ8zWwIQMNui7NUvyIO_AgKo0Th05zDb-CtWQ43X4&e=>. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss.
