Hello All, I wanted to give you an update. First, I’ll provide a <TLDR> -We (UNC-CH) seem to be in a stable situation. Clients are connecting and staying connected. STM is consuming reasonable levels of resources. -We are blind. In addition to disabling the MM to MC communication, we had to disable all SNMP. We also have a bit of automation that utilizes SNMP, and that is also broken. -There were TWO separate issues, and I’ll break those out below. It’s important to understand that the ARP issue is a separate issue from the STM issue. In our first couple of days working with Aruba, even TAC did not recognize our symptoms as two separate issues, and this is probably the thing that frustrated me the most. </TLDR>
Issue number one: ARP Our users complained they could not connect to the network and access resources for the first five to ten minutes of a class. Apple IOS devices running version 14, and some Lenovo devices ARP their entire subnet after joining Wi-Fi. The Aruba controllers have security rules in place to prevent ARP flooding and DDOS attacks utilizing ARP. As clients joined the network before classes the devices would ARP the subnet, and once a threshold was reached, the controller would begin discarding ARP packets for all clients on that controller. The result was that devices would connect, get assigned an IP via DHCP, and then ARP to get the MAC of their default gateway. That packet would be discarded, and until the controller again allowed ARP to pass, clients weren’t able to find their gateway. Depending on the client, this usually resulted in them again restarting the 802.11 join process. [Christopher Johnson, this is the behavior you are experiencing.] You can see if you are being affected by running: show datapath bwm table and checking for contract 9 (ARP). You can also check this more specifically by running: show datapath bwm type [type] contract 9 In our case, the full command was: show datapath bwm type 0 contract 9 When we first addressed this issue, we had over 2 million drops (policed) packets on each controller. Our default configuration was 992pps. After consideration, we raised our rate to 9792 expecting that multiple clients will likely be ARPing the network at the same time and recognizing how large the subnet is… and hey, it seemed like a good idea. Since then, we average less than 1-3K drops at any given time, and our users are telling us they can connect and access the network on the first try. We have seen no other detrimental effects of this change. NEXT – STM We disabled our connections between the MM and MC’s and restarted all controllers by controller cluster groups to ensure AP’s and Clients would stay connected. Once everything was restarted, we waited for students to migrate from ResNET to our Main Campus cluster. We began getting the first complaints around 10am. After checking load distribution, we found that we had even distribution of AP’s across our 8 MC’s, but 90% of our clients were connected to only two of our eight controllers in that cluster despite our load balancing configuration. This continued to be an issue, and TAC confirmed that we were appropriately configured to load balance clients at 10%. Despite disabling the MM to MC connections, we still had very high utilization by STM, and TAC decided controllers were unable to balance client connections due to that state. The next step was to block SNMP on the controller firewalls. As you can all imagine, this was a difficult decision for us, but if clients can’t connect to Wi-Fi, we don’t need SNMP to tell us it’s down…the users do a great job of that! 😊 Once we disabled SNMP, STM processor usage fell to ~30-70% and clients began balancing appropriately across controllers. So, as I said in my TLDR, we are flying blind, but user reports are coming in that the issue is much improved. Now we wait for Aruba to deliver our bug fix, and a bit of time for testing to ensure we don’t cause more issues. I want to pause here and express my second large frustration with the situation. For the affected cluster, we are running eight 7240XM controllers, which according to Aruba should support 32K clients each, yet those two controllers were incapable of load balancing due to high STM utilization when each had only 8K clients. Like many who have spoken up, we begin seeing issues as soon as client counts on a controller exceed 5K clients. I shudder to think what our experience would have been if we had half as many controllers in the cluster. Marketecture != good design JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill [email protected]<mailto:[email protected]> From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]> on behalf of James Andrewartha <[email protected]> Date: Saturday, September 11, 2021 at 9:49 PM To: [email protected] <[email protected]> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm not too familiar with how Aruba handles arps, does it do proxy arp? I have seen Apple devices go to sleep before all broadcast/multicast traffic is sent by the AP, although that was 5 years ago. So I can believe that a behaviour change could cause increased ARPs if the devices aren't seeing them. Sent from my Galaxy -------- Original message -------- From: "Turner, Ryan H" <[email protected]> Date: 12/9/21 09:16 (GMT+08:00) To: [email protected] Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We actually are allowing MORE ARPs. Apparently when policing kicks in, all connections are affecting. It can cause clients to freeze/not connect. So we actually turned the knob in the opposite direction. We were seeing counters to what amounts to large quantities of controllers pauses when the ARPs went over an arbitrarily set number. Our wireless architect can reply with the details. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 11, 2021, at 12:32 PM, Enfield, Chuck <[email protected]> wrote: HI Ryan, When you say that you detuned ARP policing, do you mean that the ARP policing on the underpinning network is now more aggressive (aka, dropping more ARP?) I ask because I’ve been wondering why we aren’t seeing this problem when other schools that made the same changes we did still are. We upgraded our underpinning network over the summer, and we’re dropping way more ARP than we were on the old network. Your post just made me realize that may be protecting our controllers. We’ve been considering changes, but we switched to an EVPN/VxLAN architecture. We’re not completely sure what the consequences of this ARP policing is, so we’ve been holding off any changes. If you had to police more aggressively to solve your problem, then we won’t start experimenting with out policers. Thanks, Chuck From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]> On Behalf Of Turner, Ryan H Sent: Saturday, September 11, 2021 10:12 AM To: [email protected] Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We had to make major changes to bring stability to Khrushchev environment. I think we have at this point. We had to significantly detune the ARP policing policies. We had to block virtually every SNMP poller. We had to reboot our controllers. We had to put in place an ACL to block communication from the Mobility masters. A ridiculous amount of work to basically get us where we were 2 years ago and we probably have 15% lower connections compared to then. I am hoping that the upcoming firmware fix will allow us to at least reverse the ACL and SNMP pollers. At this point we are pretty blind into information on individual connections. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 10, 2021, at 4:25 PM, Johnson, Christopher <[email protected]<mailto:[email protected]>> wrote: I haven’t heard anything as of yet. Although interestingly while doing a packet-capture to monitor arp/dhcp rates – noticed one client sending DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating constantly – and from the received signal strength of the client – there didn’t appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. So I’m wondering if that’s not an isolated behavior. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D&reserved=0> and Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D&reserved=0> From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> On Behalf Of Viou, Robert Sent: Friday, September 10, 2021 10:28 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Some people who received this message don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification> [This message came from an external source. If suspicious, report to [email protected]<mailto:[email protected]>] In regards to: > Aruba believes this is the cause of the new iOS operating system. Our > environment is extremely heavy iOS. We are talking to them now and will > assess the change. Has Aruba mentioned to anyone or has anyone heard any more on this being an IOS issue? Bob From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> On Behalf Of Norton, Thomas (Network Operations) Sent: Wednesday, September 1, 2021 11:31 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Thanks for the update Ryan T.J. Norton Wireless Network Architect Network Operations (434) 592-6552 [http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since ________________________________ From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> on behalf of Turner, Ryan H <[email protected]<mailto:[email protected]>> Sent: Wednesday, September 1, 2021 12:27 PM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) All, Aruba believes this is the cause of the new iOS operating system. Our environment is extremely heavy iOS. We are talking to them now and will assess the change. Ryan From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> On Behalf Of Norton, Thomas (Network Operations) Sent: Wednesday, September 1, 2021 12:17 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) ha right! Chuck beat me to it, but our limits are similar, we do allow quite a bit more before committing the action the arp rate. T.J. Norton Wireless Network Architect Network Operations (434) 592-6552 <image001.jpg> Liberty University | Training Champions for Christ since ________________________________ From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> on behalf of Enfield, Chuck <[email protected]<mailto:[email protected]>> Sent: Wednesday, September 1, 2021 12:13 PM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Thanks to firewalls, ARP is the new ping. From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> On Behalf Of Norton, Thomas (Network Operations) Sent: Wednesday, September 1, 2021 12:13 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Thats what we have seen as well, and the larger the subnet, the worst it gets. We were seeing upwards of 10k request a minuet at one point. T.J. Norton Wireless Network Architect Network Operations (434) 592-6552 <image001.jpg> Liberty University | Training Champions for Christ since ________________________________ From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> on behalf of Laramie Combs <[email protected]<mailto:[email protected]>> Sent: Wednesday, September 1, 2021 12:09 PM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Thanks TJ - it looks like the client is just walking their vlan, with excessive arp traffic, and thats having a bad effect. <image002.png> On Wed, Sep 1, 2021 at 12:07 PM Norton, Thomas (Network Operations) <[email protected]<mailto:[email protected]>> wrote: Hey Laramine/Chuck, The ARP issue most likely the Lenovo Vantage software or IOS 14. Another option outside of filtering is to enable prohibit ip spoofing and arp spoofing. T.J. Norton Wireless Network Architect Network Operations (434) 592-6552 <image001.jpg> Liberty University | Training Champions for Christ since ________________________________ From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> on behalf of Laramie Combs <[email protected]<mailto:[email protected]>> Sent: Wednesday, September 1, 2021 11:57 AM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) HEy Chuck - would you mind sharing that arp limiting client filter with me? We are seeing some new traffic patterns where it looks like user devices are just walking their subnets, and arping for everything -Laramie On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck <[email protected]<mailto:[email protected]>> wrote: We’ve seen the CPU problem, but I don’t think it resulted in Auth problems here. It may have and we just missed it because the more severe problems it caused masked them. BTW, in our case reducing the amount of ARP calmed the CPU. We applied a filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless clients could send and it smoothed out the spikes. From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> On Behalf Of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I’m hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has “identified the issue and is working on a fix.” I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We’re seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren’t happening). Our SE also said if you’re experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. <image003.jpg> -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv <[email protected]<mailto:[email protected]>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: [email protected]<mailto:[email protected]> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services [email protected]<mailto:[email protected]> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621428072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wqYqn8FAo5FPxn3WZgkFSQcskDulEgMUSyGponlIWA0%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621428072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wqYqn8FAo5FPxn3WZgkFSQcskDulEgMUSyGponlIWA0%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621438045%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gaN5TitL%2BfVcLQOC3S%2FqpnZ66%2BBwctUR5zqhp7DNyZ0%3D&reserved=0> -- Laramie Combs Network Infrastructure and Control Systems (office) 828-262-6945 (email) combslm - at - appstate.edu<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fappstate.edu%2F&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621448040%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D46z9NzEim8dkkeXlL0Y3TMenkMC7D6paKmug9cSiZc%3D&reserved=0> (*) ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621448040%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=v1kh9phKZ1mqiAF0Ymhw%2FtdcQcVnfKhfF3V14je2I6E%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621458034%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kx2xOpglNvFJ8weW2bWqQ%2Fy7L81iqyFoim09SiTM0uM%3D&reserved=0> -- Laramie Combs Network Infrastructure and Control Systems (office) 828-262-6945 (email) combslm - at - appstate.edu<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fappstate.edu%2F&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621468029%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=A4Yq8VY8ZXsJn3CCHy9Vx0E52rxvVmbI%2BwSYb5h2dTE%3D&reserved=0> (*) ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621468029%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=s21E6gK8tJJ0eXYW2kJ8gnWzheD9PgPT%2Fg6IFKfAdUU%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621478024%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=15eEkOmicytRMNeCEFzo5SscR%2FwwAjJXUYfqrH39DyM%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621478024%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=15eEkOmicytRMNeCEFzo5SscR%2FwwAjJXUYfqrH39DyM%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621488022%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RCxgPeAFo1hedKZqRPS4mM%2FMJnHMnoz2ohnqKfcK3sg%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621488022%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RCxgPeAFo1hedKZqRPS4mM%2FMJnHMnoz2ohnqKfcK3sg%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621498013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Tbq6%2FCkFTLTXFSu6nUin%2FPwxt5qGuukJ%2FkrjNkPBZD8%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621508008%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4FtCev5brRHaULYVe7fIuLUVbo7Vps3w6A0OUK22HrI%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621508008%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4FtCev5brRHaULYVe7fIuLUVbo7Vps3w6A0OUK22HrI%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621518007%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bq%2B%2Bk3u3LPVXMlcnj2MxScvd%2FW8mpEknCVhGleJFXks%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
