There were 2 things that I always did when modifying the service registry in CAS v3.x.
1) After making a change, I ran a short python script that would validate the well formedness of the JSON and doe some other simple checks. 2) I would put the service URL in my browser to see what our login page looked like (we display different UI elements based on the service). #1 took care of most issues related to typos. #2 was also a good sanity check for new services-- if I botched it somehow, I would our "please contact the help desk" page instead of the service page. In CAS v5, the service definitions are now broken out, so I may need to revisit a validation script of some kind. Thanks, Carl Waldbieser ITS Identity Management Lafayette College ----- Original Message ----- From: "Bryan Wooten" <ttbaja...@gmail.com> To: cas-user@apereo.org Sent: Tuesday, May 2, 2017 8:41:07 PM Subject: Re: [cas-user] bad json service definition breaks cas 5.0.x? You are not wrong. We restart CAS once a week. We are currently running 3.6.x, it has a memory leak with load over time. We do 400k logins per day. (CPU goes to 100% doing nothing but garbage collection) The weekly restart mitigates / solves (?) this issue, But if in the interval we messed up the JSON Service Registry then the CAS server is not working. So yes perhaps an enhancement request for better notification of a bad Service Registry re-load may be warranted. I think a proper log4.xml setting in conjunction with monitoring like Solarwinds/Orion or even ELK would solve this issue with a timely notification to the correct people. So many moving pieces. We do the best we can to support our organization. On Tue, May 2, 2017 at 5:48 PM, Baron Fujimoto <ba...@hawaii.edu> wrote: > Yes, but one of the nice things about CAS was that we could drop new > service definitions into a running instance and have them picked up > automatically. Anything that requires us to actually restart our service > get us involved in a formal change management process and restricts us to > weekend early mornings. :( > > The thing that seems inconsistent to me in this case, is that the bad > service definition doesn't tank an already running instance. The faulty > definition just doesn't get incorporated into the running config. This > seems much preferable to breaking everything should the system be > restarted for some reason, IMO. > > So, I guess I understand the way it is, but must it be that way? Should > it be that way? > > On Mon, May 01, 2017 at 07:20:00PM -0600, Bryan Wooten wrote: > >Ok, If you are manually editing the JSON service registry or using any > tool > >(home grown or provided by Apereo) you MUST carefully validate the final > >JSON file syntax. > > > >I never make a change during normal hours. > > > >We are HA with 4 CAS servers. I make the change on one server. (They are > >all standalone no rsync or cross mounts for any config file) I then > restart > >that one server and verify is comes back up and starts servicing tickets. > >(All CAS servers are behind a Citirix Netscalar Load Balancer). So there > is > >no outage. > > > >After I confirm the change has the correct JSON syntax I then push the > >change to the other servers. > > > >Owning SSO is terrifying. We can never have an outage for 60k users and > >500 servers. Never. Or my life gets bad. > > > >So yeah, it is manual and I take great care and someday I hope to put in > >place an automated system. > > > >Until then I live with a little stress..... > > > >On Mon, May 1, 2017 at 6:29 PM, Baron Fujimoto <ba...@hawaii.edu> wrote: > > > >> So this happened. We are using CAS 5.0.3.1 with JSON service registry > >> files. We inadvertantly created a service registration with an error in > >> the regex for the serviceID (improperly escaped "\\." as "\."). Although > >> we didn't notice it at the time, this was logged with the error/warnings > >> > >> ERROR [org.apereo.cas.services.AbstractResourceBasedServiceRegistryDao] > - > >> <Error reading configuration file Bad_serviceID-20170328162739.json> > >> java.lang.IllegalArgumentException: org.hjson.ParseException: Expected > >> valid escape sequence at 3:30 > >> ... > >> WARN [org.apereo.cas.services.AbstractResourceBasedServiceRegistryDao] > - > >> <Could not load service definition from file /home/cas/cas/services/Bad_ > >> serviceID-20170328162739.json> > >> WARN [org.apereo.cas.services.AbstractResourceBasedServiceRegistryDao] > - > >> <1 errors encountered when loading service definitions. New definitions > are > >> not loaded until errors are corrected> > >> > >> But CAS continued on working in general. However, some time later, the > CAS > >> service was restarted after an OS reboot. Upon restart, the same errors > >> are noted, but there was crucial difference > >> > >> INFO [org.apereo.cas.services.DefaultServicesManagerImpl] - <Loaded 0 > >> services from JsonServiceRegistryDao.> > >> > >> None of the other valid service definitions were loaded, and thus our > CAS > >> was effectively broken for everyone by this one bad definition. > >> > >> Is there a way to configure CAS to just ignore the bad definitions > rather > >> than just fail completely in situations like this? While we probably > should > >> have caught this sooner, it definitely turned out to be a time bomb for > >> us. Is there a good reason for this behavior that we're missing? > >> > >> Aloha, > >> -baron > >> -- > >> Baron Fujimoto <ba...@hawaii.edu> :: UH Information Technology Services > >> minutas cantorum, minutas balorum, minutas carboratum desendus pantorum > >> > >> -- > >> - CAS gitter chatroom: https://gitter.im/apereo/cas > >> - CAS mailing list guidelines: https://apereo.github.io/cas/ > >> Mailing-Lists.html > >> - CAS documentation website: https://apereo.github.io/cas > >> - CAS project website: https://github.com/apereo/cas > >> --- > >> You received this message because you are subscribed to the Google > Groups > >> "CAS Community" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to cas-user+unsubscr...@apereo.org. > >> To view this discussion on the web visit https://groups.google.com/a/ > >> apereo.org/d/msgid/cas-user/20170502002914.mfo3dcbcul3oo5k5% > >> 40combobulate.mgt.hawaii.edu. > >> > > > >-- > >- CAS gitter chatroom: https://gitter.im/apereo/cas > >- CAS mailing list guidelines: https://apereo.github.io/cas/ > Mailing-Lists.html > >- CAS documentation website: https://apereo.github.io/cas > >- CAS project website: https://github.com/apereo/cas > >--- > >You received this message because you are subscribed to the Google Groups > "CAS Community" group. > >To unsubscribe from this group and stop receiving emails from it, send an > email to cas-user+unsubscr...@apereo.org. > >To view this discussion on the web visit https://groups.google.com/a/ > apereo.org/d/msgid/cas-user/CAG9x2GVK0zMcb2Swb0KhKK6n% > 3D4pG%3DiZFnu8aRLtdxb%2BtJN7%2B3Q%40mail.gmail.com. > > -- > Baron Fujimoto <ba...@hawaii.edu> :: UH Information Technology Services > minutas cantorum, minutas balorum, minutas carboratum desendus pantorum > > -- > - CAS gitter chatroom: https://gitter.im/apereo/cas > - CAS mailing list guidelines: https://apereo.github.io/cas/ > Mailing-Lists.html > - CAS documentation website: https://apereo.github.io/cas > - CAS project website: https://github.com/apereo/cas > --- > You received this message because you are subscribed to the Google Groups > "CAS Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to cas-user+unsubscr...@apereo.org. > To view this discussion on the web visit https://groups.google.com/a/ > apereo.org/d/msgid/cas-user/20170502234855.3tewdzzz6buc4mjh% > 40combobulate.mgt.hawaii.edu. > -- - CAS gitter chatroom: https://gitter.im/apereo/cas - CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html - CAS documentation website: https://apereo.github.io/cas - CAS project website: https://github.com/apereo/cas --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscr...@apereo.org. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAG9x2GXfierNrnA3yFd6dsx35XOdQOreCHyApa_ysJ4WA4ooGw%40mail.gmail.com. -- - CAS gitter chatroom: https://gitter.im/apereo/cas - CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html - CAS documentation website: https://apereo.github.io/cas - CAS project website: https://github.com/apereo/cas --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscr...@apereo.org. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/1577765562.11317473.1493816921191.JavaMail.zimbra%40lafayette.edu.
