Hi Keith I wish Id seen your mail earlier as we had been running 8.4.x series code for a while now and I could have grabbed more meaningful stats. We have just moved to 8.5.0.4 this weekend so I can only provide you with 2 days’ worth of information. So far we have a stable cluster with no disconnects and little to no AP bootstraps.
I will continue to monitor this and will let you know if anything changes. M From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Miller, Keith C Sent: 07 December 2019 01:56 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] [WIRELESS-LAN] ArubaOS 8.x cluster disconnects Hi T.J., We are not running port-channels directly to our MDs, but we do have port-channels on upstream switches. Based on some things I’ve seen in the packet captures we’ve collected and sent to Aruba (IPSec sequence numbers out of order), I’ve considered the fact that some packets might be load-balanced down different links and are arriving out of order, but the load balancing algorithms in place and traffic profile do not support that theory. We do not run CPSec in our environment, but thank you for the bit of info. It depends on the environment, but it ranges from in the 10s to as high as 86 on 1 AP since the last image upgrade. I’d say the average in our most frequently disconnecting cluster is somewhere in the 30-40s. Here’s a couple of examples: AP Counters ----------- Name AP Boots Acked Bootstraps (Total) Reboots MEJ_1207A 4 (54 ) 21 MEJ_1207B 3 (44 ) 22 Regards, Keith From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of "Norton, Thomas (Network Operations)" <tnort...@liberty.edu<mailto:tnort...@liberty.edu>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Friday, December 6, 2019 at 8:37 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] [WIRELESS-LAN] ArubaOS 8.x cluster disconnects Hey Keith, We’re running 8.3.0.10 with multiple clusters and are not running into any issues on our end. Our cluster statistics are fairly clean other than some issues on some of our switches that we have been running into. One question, are you running port channels to your mds? If so, we have run into issues in the past with sending fast pdus, causing our links to flap. Another thing is cpec, if your running it, highly recommend jumbo frames due to the extra overhead on the management tunnels. This is still something we’re working to implement internally. Out of curiosity, when you run the counters command how many bootstraps are you seeing per ap on average? T.J. Norton Wireless Network Architect Network Operations Office: (434) 592-6552<tel:(434)%20592-6552> [Image removed by sender. http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since 1971 On Dec 6, 2019, at 7:52 PM, Miller, Keith C <keith.mil...@unc.edu<mailto:keith.mil...@unc.edu>> wrote: ________________________________ [ EXTERNAL EMAIL: Do not click any links or open attachments unless you know the sender and trust the content. ] ________________________________ Hello all, As many of you know, we’re an Aruba shop and we’re running multiple versions of 8.x in our environment. We are also a Nyansa Voyance customer and for those who are also Nyansa customers will probably remember back in October when they changed the default behavior for AP down/reboot events from “No Priority” to “Always P2”. Almost immediately, we began receiving alerts from Voyance about large amounts of APs going down at the same time. After looking at our controllers and other NMS tools, we realized that the APs were not actually going down, but the radios on the APs were rebootstrapping. For those unfamiliar with what rebootstrapping is, it essentially means that the radios of the AP rebooted, but the AP itself stayed up. This is typically caused by missed heartbeats and/or when an AP reconnects to a controller. In a clustered environment, when a controller fails, an AP should gracefully move to its S-AAC with little to no impact. However, in our case we were seeing APs not gracefully failover after missing heartbeats and this was causing the rebootstraps. This impacts clients and our users so obviously we were very concerned with what we had found. After opening a case with Aruba TAC, we discovered that the cluster members were disconnecting from each other. You can see if this is happening in your environment by running the “show lc-cluster heartbeat counters” command on one of the MDs in a cluster. You’re looking for the last column that indicates the last time of disconnect. For us, this has been occurring in multiple environments (8.3, 8.4, and 8.5) at least since we began looking into it back in October. We’ve sent many logs, traces, and now packet captures to the Aruba TAC team. At the request of TAC, we’ve changed heartbeat thresholds and enabled BCMC optimization on VLAN interfaces even though we have it enabled at the SSID level. While some of these efforts have slowed down the frequency of the disconnects, they are still occurring. So I’m looking to get some feedback from those that are running AOS 8.x in their environment. Are you seeing this problem in your environment? Lastly, if you’re experiencing this issue or you’re just interested in finding out more about the health of your environment, you can also verify if you have APs that are rebootstrapping with the “show ap debug counters” command. If you want to isolate a particular AP and gather more information, you can run the “show ap debug system-status ap-name” command. Here’s what it looks like when the AP doesn’t gracefully failover: Cluster Failover Information ---------------------------- Date Time Reason (Latest 10) -------------------------------------- 2019-11-25 01:10:20 Delete A-AAC:172.27.xx.xx, cluster enabled=1. fail-over to 172.27.xx.xx, sby status=1 Thanks in advance for any and all feedback. Regards, Keith C. Miller Wireless Architect, ITS Comm. Technologies University of North Carolina Chapel Hill O: (919)962-6564 M: (803)464-2397 | keith.mil...@unc.edu<mailto:keith.mil...@unc.edu> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=02%7C01%7Ctnorton7%40LIBERTY.EDU%7C7c47124f3eba4466ac6908d77aafc316%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C637112767631163431&sdata=VkwZrOQyqmazD7NbJfuDWDtdZW6HwjD2c3ax1ZbJ8PM%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
