Thank you for the good thoughts on the storm. Luckily we are fine. So far we’ve been told that the issue we experienced was a combination of two things: 1) the 8540’s memory queues and buffers reached their maximum capacity. This affected both 802.1X and CAPWAP. Thus the AP flapping. 2) RADIUS and EAP timers must be EXTRA optimized. I say EXTRA, because we’ve always followed best practices and recommendations from TAC.
This is a good document to read: https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/118703-technote-wlc-00.html Finally, what is most interesting is the fact that even though the 8540 is advertised to support 6000 APs and 64000 clients, these numbers do not seem to be valid if your environment is mainly 802.1X. So, if your environment is mainly 802.1X, and you have an 8540, I would recommend you talk to your Cisco SE so they can tell you what the official supported number of APs is. I’ve yet to find any official documentation that even hints to this. Miercom performed a comparative test in 2015 between Aruba and Cisco, and in the report they did test client authentication rate, but only for the Cisco 5520. https://www.cisco.com/c/dam/en/us/products/collateral/wireless/8540-wireless-controller/miercom-report-wlcs-cisco-aruba.pdf TAC’s recommendation is for us to use 8.2.160 on the 8540s. We will make all necessary config changes and start moving APs in waves of 500 slowly so we can watch utilization. Our plan also includes not to exceed the AP capacity of the 8540s by 50%-60%. If this works, we will have to get an additional pair of 8540s. I’ll let you know if we are successful. BTW, we require to have AVC turned on. TAC is very concerned about this. We’ll also be watching this. -Hector From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Lee H Badman Sent: Wednesday, August 30, 2017 6:43 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Move In/Opening Week- Any Problems? Hi Hector, I hope the storm is not causing havoc for you down there- good thoughts to you on that. Did you get anywhere with Cisco on your 8540/8.2.160 problems? I'm being told we may need to go that same combination and it doesn't inspire confidence. Evidently my 8.2.151 (you know... one of those STABLE code versions) may be a time bomb that caused a spontaneous 8540 reboot. The comment was made that our 3300 APs on a platform that supposedly supports 6000 somehow equals a dense deployment and that we likely are hitting: ___ Regarding the logs, I was able to check the logs, and yes It seems your deployment is a high-density deployment with over 3000 APs. Based on your deployment and the logs I was able to identify this It seems the WLC is having load process utilization on the task SpamReceive Task and HAConfigSyncTask. spamApTask1 5992 ( 53/ 78) 0 ( 0/ 0)% 30 22 spamApTask0 5991 ( 72/ 70) 0 ( 0/ 0)% 30 5 spamReceiveTask 5990 ( 52/ 78) 0 ( 0/ 0)% 99 0 spamSocketTask 5989 (175/ 32) 0 ( 0/ 0)% 0 13 HAPeerToPeerCommTa 5988 ( 90/ 64) 0 ( 0/ 0)% 0 7 rmgrPing 5987 ( 80/ 67) 0 ( 0/ 0)% 0 13 HAConfigSyncTask 6204 (240/ 7) 0 ( 0/ 0)% 99 3 Based on the symptoms, the WLC version and your WLC density. You may be hitting bug. CSCvd20251 - Data Plane stopped working on Cisco 5508 WLC running 8.0.140.0<https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd20251/?reffering_site=dumpcr> ___ I hope to have confirmation today. I can't imagine what Cisco could have done between .151 and .6 to make this sort of thing better, and I am really interested in whether they isolated your own .160 problems. There is no way in hell I'm moving to that version without seeing case notes on every single issue people are having in this continual cycle of trading one set of bugs for another. This game just isn't fun anymore. Thanks- Lee Badman | Network Architect | CWNE #200 Information Technology Services 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 f 315.443.4325 e lhbad...@syr.edu<mailto:lhbad...@syr.edu> w its.syr.edu SYRACUSE UNIVERSITY syr.edu ________________________________ From: The EDUCAUSE Wireless Issues Constituent Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Hector J Rios <hr...@lsu.edu<mailto:hr...@lsu.edu>> Sent: Friday, August 25, 2017 3:11 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Move In/Opening Week- Any Problems? Here’s ours: 2 8540s in HA mode (bought with the idea of replacing all WiSM2s) 4 pairs of WiSM2s in HA mode 3 server ClearPass cluster for both eduroam and guest Main SSID: eduroam with PEAP/MSCHAP Mix of WAPs; 3500, 3600, 3700, 2800, 1810w Total number of WAPs: 3500 21000 peak users We tested the 8540s extensively over the spring and summer, primarily with the 8.2.151 code and a mix of 2800s and 1810ws. We had AVC turned on, and were using RLANs for the wired ports. The largest number of WAPs we had on this pair was 469. We tested code 8.2.160 towards the end of the summer with all WAPs on the 8540s, and had no issues. First the day of classes, we had all WiSM2s running 8.2.160 simply as a backup. Early morning we started getting reports of 802.1X authentication failures (these failures had nothing to do with ClearPass). Shortly after that, WAPs starting flapping (disconnecting from the 8540s moving to WiSM2s and then moving back again). We tried playing with the TCP MSS setting, adjusting EAP timers, turning AVC off and multiple other things, but nothing worked. In the end, we downgraded the WiSM2s to 8.0.140 and moved all WAPs that were not 2800 or 1810s. The 8540s were downgraded to 8.2.151 so the 2800s and 1810s would have a controller to connect to. Network stability was restored after this. Needless to say it was a very unpleasant experience. We are still working with Cisco to find out the root cause of the problem. Hector Rios Louisiana State University From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Lee H Badman Sent: Friday, August 25, 2017 8:22 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Move In/Opening Week- Any Problems? It might be beneficial to share notes in case other schools are hitting common problems. I’m wondering how everyone who is in the thick of it is faring with back-to-school? On this end, we are doing OK halfway to our expected total daily peak clients (we’re at 15K now high water mark). Our significant WLAN-related changes since end of Spring semester • Running 8.2.151 on our 8540s • Significant quantities of Wave 2 APs • ISE as RADIUS (only, no NAC, no onboarding) No changes to: • our guest WLAN (Clearpass/an Aruba controller pair) • onboarding (Cloudpath Wiz) • overall topology • open network in dorms for gadgets • non-use of AVC, it crapped out and never got solved after hundreds of hours with TAC Fears: • We haven’t yet hit the scale that will reveal problems with any of the newer stuff listed above Anyone else care to share? -Lee Lee Badman | Network Architect Certified Wireless Network Expert (#200) Information Technology Services 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 f 315.443.4325 e lhbad...@syr.edu<mailto:lhbad...@syr.edu> w its.syr.edu SYRACUSE UNIVERSITY syr.edu ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss.
