Dan, Do you have the APs in public subnets or private subnets? Occasionally we see this problem happening in our environment as well. Currently we put APs in public subnets. I worked with Cisco TAC on this and we could not find anything. Finally TAC suggested we move APs to private subnets and we are considering that.
--- Dennis Xu Network Analyst, Computing and Communication Services University of Guelph 5198244120 x 56217 ----- Original Message ----- From: "Dan Brisson" <dbris...@uvm.edu> To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Sent: Wednesday, February 1, 2012 12:31:02 PM Subject: Re: [WIRELESS-LAN] Cisco APs losing CAPWAP session Good question. Turns out the APs use UTC time, which appears to be correct: AP#sh clock *17:29:03.737 UTC Wed Feb 1 2012 -dan Dan Brisson Network Engineer University of Vermont (Ph) 802.656.8111 dbris...@uvm.edu On 2/1/2012 12:11 PM, Craig Eyre wrote: > Dan, > > What time is your controller showing? From your log messages it looks like > its 2:16pm there? AP's have issues when the time is off between the > controller and the ap's. > > > Craig Eyre > Network Analyst > IT Services Department > Mount Royal University > 4825 Mount Royal Gate SW > Calgary AB T2P 3T5 > > P. 403.440.5199 > E. ce...@mtroyal.ca > > "The difference between a successful person and others is not a lack of > strength, not a lack of knowledge, but rather in a lack of will." Vincent > T. Lombardi > > > > > From: Dan Brisson<dbris...@uvm.edu> > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Date: 02/01/2012 09:32 AM > Subject: Re: [WIRELESS-LAN] Cisco APs losing CAPWAP session > Sent by: The EDUCAUSE Wireless Issues Constituent Group Listserv > <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> > > > > This what we see on the AP at the time the AP disjoins: > > *Feb 1 14:16:25.174: %DTLS-5-SEND_ALERT: Send FATAL : Close notify > Alert to 10.246.207.214:5246 > *Feb 1 14:16:25.227: %WIDS-5-DISABLED: IDS Signature is removed and > disabled. > *Feb 1 14:16:25.227: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY > *Feb 1 14:16:25.227: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY > *Feb 1 14:16:25.293: %LINK-5-CHANGED: Interface Dot11Radio0, changed > state to administratively down > *Feb 1 14:16:25.293: %LINK-5-CHANGED: Interface Dot11Radio1, changed > state to administratively down > *Feb 1 14:16:25.299: %LINK-5-CHANGED: Interface Dot11Radio0, changed > state to reset > *Feb 1 14:16:25.309: status of voice_diag_test from WLC is false > *Feb 1 14:16:25.309: %LINK-3-UPDOWN: Interface Dot11Radio1, changed > state to up > *Feb 1 14:16:25.318: %LINK-3-UPDOWN: Interface Dot11Radio0, changed > state to up > > -dan > > Dan Brisson > Network Engineer > University of Vermont > (Ph) 802.656.8111 > dbris...@uvm.edu > > > On 2/1/2012 10:30 AM, Mike Goebel wrote: >> Dan, have you tried logging into the AP itself and checking the logs >> by chance? >> >> Mike >> >> On 2/1/2012 10:03 AM, Dan Brisson wrote: >>> It does seem as though I've grabbed some folks attention. I sure hope it >>> turns out to not be something simple. :) >>> >>> I could certainly try moving the APs around...easy enough to do, >>> although from what we've seen, the pattern of AP drops is so totally >>> random, hard to say if I'll see anything. At this point though, it's >>> worth a shot. >>> All interfaces clean and no QoS in place. >>> >>> Not sure if this will come through for everyone, but here's an example >>> of what I see after an AP drops. This is from the controller, on the >>> General tab for an AP: >>> >>> >>> >>> Thanks, >>> -dan >>> >>> Dan Brisson >>> Network Engineer >>> University of Vermont >>> (Ph) 802.656.8111 >>> dbris...@uvm.edu >>> >>> >>> On 2/1/2012 9:26 AM, Garry Peirce wrote: >>>> I think you have some of us all getting curious! ;-) >>>> >>>> Could you put a historically stable admin AP onto the 5508 and >>>> vice-versa to >>>> see if behaviors change? >>>> Do we assume that all switchports in the path are showing they're >>>> running >>>> clean? >>>> Any QoS config in place on the switches? >>>> >>>> >>>> -----Original Message----- >>>> From: The EDUCAUSE Wireless Issues Constituent Group Listserv >>>> [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Dan Brisson >>>> Sent: Wednesday, February 01, 2012 9:09 AM >>>> To:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >>>> Subject: Re: [WIRELESS-LAN] Cisco APs losing CAPWAP session >>>> >>>> Good to know. >>>> >>>> The trunks are actually all 10Gig links, or 90% of them are, so >>>> utilization >>>> is most likely not the case, which I'm able to verify from Cacti >>>> graphs. >>>> The APs are connected to 3560Xs PoE switches that then uplink into >>>> either a >>>> 3560E-12D or directly into a 4900M where the 5508s are connected. >>>> Certainly >>>> can't rule out physical layer issue somewhere, although it's so wide >>>> spread >>>> across 2 different 5508s that we would need to have multiple issues. >>>> The other interesting thing for us is that the 500 or so APs on our >>>> admin >>>> side that do not lose their CAPWAP session, join to WiSMs, not 5508s. >>>> >>>> Thanks, >>>> -dan >>>> >>>> Dan Brisson >>>> Network Engineer >>>> University of Vermont >>>> (Ph) 802.656.8111 >>>> dbris...@uvm.edu >>>> >>>> >>>> On 1/31/2012 8:44 PM, Garry Peirce wrote: >>>>> We have ~1400 (1240s->3502's) running 7.0.116 and have no such issues. >>>>> >>>>> I would guess at packet loss as well - some things you might look at: >>>>> Are the trunks carrying user/AP traffic seem congested when the APs >>>>> drop? >>>>> Have you verified there are no duplex issues? It may exhibit itself >>>>> more as traffic levels rise. >>>>> ResHall switching significantly different than on the admin side? >>>>> >>>>> Probably need further topology, version, config info, but as you've a >>>>> case open, the TAC will likely ask the same and help find the >>>>> culprit(s) >>>> for you. >>>>> -----Original Message----- >>>>> From: The EDUCAUSE Wireless Issues Constituent Group Listserv >>>>> [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Dan Brisson >>>>> Sent: Tuesday, January 31, 2012 8:30 AM >>>>> To:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >>>>> Subject: [WIRELESS-LAN] Cisco APs losing CAPWAP session >>>>> >>>>> I'm curious if any Cisco users out there are experiencing or have >>>>> experienced what we're seeing on our campus. This past summer we >>>>> installed 3502i's in all of our residence halls - approximately 500 >>>>> total. Ever since the students have moved in, we will get messages >>>>> from WCS stating that "AP XYZ" is down and disassociated from the >>>>> controller. When I check out the AP, the uptime is fine, but the >>>>> "CAPWAP join time" is for like 30 seconds, or however long it took >>>>> me to >>>> check. >>>>> We've tracked this and it is totally random as to what AP will drop, >>>>> which makes troubleshooting this very tough. The log on the AP isn't >>>> helpful. >>>>> I'm working with TAC who suggests that keepalives are getting missed. >>>>> I'm not sure why that would be the case since we have another >>>>> 500 or so APs on the admin side that very rarely drop. Adding to >>>>> that, when the students left for break, the AP drops stopped. They >>>>> came back, and sure enough, the drops start up again. >>>>> >>>>> I will say that the AP always joins back immediately, but for the time >>>>> that it does drop A) I'm sure connectivity is affected in that area >>>>> and >>>>> B) we get an email. >>>>> >>>>> Anyone experiencing this? >>>>> >>>>> Thanks, >>>>> -dan >>>>> >>>>> >>>>> -- >>>>> Dan Brisson >>>>> Network Engineer >>>>> University of Vermont >>>>> (Ph) 802.656.8111 >>>>> dbris...@uvm.edu >>>>> >>>>> ********** >>>>> Participation and subscription information for this EDUCAUSE >>>>> Constituent Group discussion list can be found at >>>> http://www.educause.edu/groups/. >>>>> ********** >>>>> Participation and subscription information for this EDUCAUSE >>>>> Constituent >>>> Group discussion list can be found athttp://www.educause.edu/groups/. >>>> >>>> ********** >>>> Participation and subscription information for this EDUCAUSE >>>> Constituent >>>> Group discussion list can be found athttp://www.educause.edu/groups/. >>>> >>>> ********** >>>> Participation and subscription information for this EDUCAUSE >>>> Constituent Group discussion list can be found >>>> athttp://www.educause.edu/groups/. >>> ********** Participation and subscription information for this EDUCAUSE >>> Constituent Group discussion list can be found at >>> http://www.educause.edu/groups/. >>> > ********** > Participation and subscription information for this EDUCAUSE Constituent > Group discussion list can be found at http://www.educause.edu/groups/. > > ********** > Participation and subscription information for this EDUCAUSE Constituent > Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
