The controller has a console port connected via access server. Also the CIMC interface was accessible and working. TAC was able to check the controller and the uplink switch while the issue was happening but couldn't find anything conclusive. It seems like the controller was just confused. Your heaviest loaded 8540 seems to have similar load with ours, the only major difference I see is the AP model.
Thank. Dennis Sent from Samsung Mobile -------- Original message -------- From: James Helzerman Date:09-23-2017 11:10 AM (GMT-05:00) To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] 8540 Controller load related issue? Hi. Did you have out of band access to the controller via the service port or a terminal server? Curious if it was networking related or if the controller got confused. We run 8 pairs of 8540 controllers running 8.2.160 and a few 8.2.150 that are in the progress of being upgraded. Out heaviest loaded controller is about 3000 to 3500 2702 APs and maybe a few 1810w. The client load has been up around 10k - 13k peak with 90% being 802.1x. so far we have not seen this issue but have had two controller crashes with crash logs. In both cases HA worked as expected with little to no impact to the users. Jimmy University of Michigan On Sep 22, 2017 11:32 PM, "Dennis Xu" <d...@uoguelph.ca<mailto:d...@uoguelph.ca>> wrote: Thank you Jess for your information, but not quite the same as our situation. In our case, not just the APs are disassociated, the controller can't be pinged from anywhere, nor the controller can ping it's default gateway. The controller was totally lost from the network. Its uplink port channel stayed up. After reboot, it regained network connectivity. Dennis Sent from Samsung Mobile -------- Original message -------- From: Jess Walczak Date:09-22-2017 9:22 PM (GMT-05:00) To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] 8540 Controller load related issue? Dennis, We are running an HA pair of 8510's with 8.0, so not quite the same, but this week we had the same thing you described happen, where more than half the APs suddenly disassociated but then came right back up (after a little back and forth since they found an older controller running newer code on an ordinally lower IP). We traced the event to an influx of traffic from China to whatever devices happen to be on our gaming vlan (which is several hundred), which we setup such that unlike all other wireless on the campuses, it shares an IP space with a wired vlan, and those IPs are some of our publicly addressable class B, so that their games won't squawk about being NAT'd. We are looking into the firewall logs to make sure there isn't something that we might need to be blocking in the future, but my working theory is the sudden massive wave of retransmits caused the APs to disassociate and reset. Thanks!--JW Jess Walczak Senior Network Analyst Information Technology Services jwwalc...@stthomas.edu<mailto:jwwalc...@stthomas.edu> University of St. Thomas | stthomas.edu<http://stthomas.edu> On Sep 22, 2017 3:17 PM, "Dennis Xu" <d...@uoguelph.ca<mailto:d...@uoguelph.ca>> wrote: This summer we added more APs to 8540 controller. Now our 8540 controller has 2350 APs(mainly 1810W and 2802 APs) and 13K (increased from 6k in March/April) concurrent 802.1X users at peak. We also upgraded the controllers from 8.2.150.0 to 8.2.160.0 during summer. Now the controller has run into an issue twice that it suddenly lost all communications from the network. It cannot be connected from anywhere but it did not crash(was up and running). All APs disassociated from it and re-associated to the secondary 8540. I suspect this issue is somehow load related although Cisco's advertised support is 64k clients. We would like to know if anyone else has more concurrent 802.1X users than us on 8540 and do not have this issue? And what code do you use? Thanks. Dennis Xu | Analyst III, Network Infrastructure Computing and Communications Services (CCS) | University of Guelph ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss.
