“First, we are not seeing ST validations being logged, ever, for any user - that may or may not be part of how the SAML authentication works, we're not sure.“
The SAML 2.0 profile used by Google Apps works that way; GA validates using the digital signature and never calls /samlValidate. “Second, the massive number of STs are being created on only one server (we can tell by the host name in the logged ST) but the OTHER SERVER is where the memory is growing out of bounds.” What do your servers use to replicate STs? Do you have that method configured to expire STs appropriately? Best regards, -- Carlos. From: David A. Kovacic [mailto:d...@case.edu] Sent: Thursday, 04 December, 2014 14:28 To: cas-user@lists.jasig.org Subject: Re: [cas-user] Rapid Memory Consumption and Interpreting Heap Dump This is becoming something of a head-scratcher now. We again saw the rapid memory consumption issue this morning. Fortunately it happened during the day and I was able to stop and start the affected server before we had run out of heap memory, so the service continued to function on the other load-balanced server. While that cost us getting a heap dump this time, since we never triggered an OoM error, we were able to use the logs to recreate some of what is going on. We have no idea however WHY the memory suddenly starts to go through the roof on the server it does (more on that in a bit), or what exactly is causing the bad behavior that triggers the climb. First some background so people can weigh in with their experience and suggestions: We are running CAS 4.0.0 on two servers in separate data centers load balanced using BigIP's F5 load balancer in an active/active configuration. "Sticky sessions" are set on the F5 with a session timeout of 5 minutes (the same as the default for the web page timeout). We are using an ehcache ticket cash replicating between the two servers (with bootstrapping turned on). Max lifetime for TGTs is 12 hours with a 6 hour idle timeout. Max lifetime for STs is 5 minutes with an idle timeout of 0. We are running staggered ticket registry cleaners on the two systems but since this does not seem to be a memory leak problem, we will likely turn those off at some point. We are using SAML 2 to do the Google authentication as described in the 4.0 documentation under the "SAML protocol" section. What we see happening: We are seeing massive numbers of repeated logins to Google via the SAML2 service by one user in a very short time frame (Monday's incident was ~6000 in about 90 minutes, today's was about 3000 in about 70 minutes). We see about 30 logins/minute. The Google audit logs show that these are actual login events. We see (catalina logs) one authentication failure followed by one authentication success and a subsequent granting of a TGT to the user. Thereafter we see thousands of STs for this user (as I said, approximately 30/minute) for the duration of the "event". During the "event" memory on one of the servers grows and even after the "event" is over, never seems to decline, so it appears that something in the heap is not being garbage-collected correctly. Here's where things get interesting: First, we are not seeing ST validations being logged, ever, for any user - that may or may not be part of how the SAML authentication works, we're not sure. Second, the massive number of STs are being created on only one server (we can tell by the host name in the logged ST) but the OTHER SERVER is where the memory is growing out of bounds. The server where the STs are actually created never seems to have the memory issues. From the one heap dump we got, it looks like something to do with the Google service is where most of the memory is being sucked up on the affected server (again the other server from where the ST is created). We log what the statistics pages report as the number of unexpired and expired TGTs and STs for each server. The numbers for each server are generated generally about 3 seconds apart and "track" with each other, being within one or two of each ticket type on each server. We are surmising that SOMETHING is being sent to the affected server via the ticket replication process (most likely something to do with the ST processing) but is either getting properly cleaned up on the server where the ST is actually created and not on the other server, or the bogus data is never actually created on the server where the ST is created and only is getting created on the affected server. This is our argumentExtractorsConfiguration.xml file: <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to Jasig under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. Jasig licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at the following location: http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <beans xmlns= <http://www.springframework.org/schema/beans> "http://www.springframework.org/schema/beans"; xmlns:xsi= <http://www.w3.org/2001/XMLSchema-instance> "http://www.w3.org/2001/XMLSchema-instance"; xmlns:p= <http://www.springframework.org/schema/p> "http://www.springframework.org/schema/p"; xmlns:c= <http://www.springframework.org/schema/c> "http://www.springframework.org/schema/c"; xmlns:util= <http://www.springframework.org/schema/util> "http://www.springframework.org/schema/util"; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd";> <description> Argument Extractors are what are used to translate HTTP requests into requests of the appropriate protocol (i.e. CAS, SAML, SAML2, OpenId, etc.). By default, only CAS is enabled. </description> <bean id="casArgumentExtractor" class="org.jasig.cas.web.support.CasArgumentExtractor" /> <!-- Needed for general SAML integration --> <bean id="samlArgumentExtractor" class="org.jasig.cas.support.saml.web.support.SamlArgumentExtractor" /> <!-- Needed for Google integration via SAML --> <bean id="googleAccountsArgumentExtractor" class="org.jasig.cas.support.saml.web.support.GoogleAccountsArgumentExtractor" p:privateKey-ref="privateKeyFactoryBean" p:publicKey-ref="publicKeyFactoryBean" p:alternateUsername="eduPersonPrincipalName" /> <util:list id="argumentExtractors"> <ref bean="casArgumentExtractor" /> <!-- Needed for general SAML integration --> <ref bean="samlArgumentExtractor" /> <!-- Needed for Google integration via SAML --> <ref bean="googleAccountsArgumentExtractor" /> </util:list> <!-- The following configure the keys needed to talk to Google via SAML --> <bean id="privateKeyFactoryBean" class="org.jasig.cas.util.PrivateKeyFactoryBean" p:location="classpath:private.p8" p:algorithm="RSA" /> <bean id="publicKeyFactoryBean" class="org.jasig.cas.util.PublicKeyFactoryBean" p:location="classpath:public.key" p:algorithm="RSA" /> <!-- End of Google SAML additions --> </beans> And our uniqueIdGenerators.xml file: <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to Jasig under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. Jasig licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at the following location: http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <beans xmlns= <http://www.springframework.org/schema/beans> "http://www.springframework.org/schema/beans"; xmlns:xsi= <http://www.w3.org/2001/XMLSchema-instance> "http://www.w3.org/2001/XMLSchema-instance"; xmlns:p= <http://www.springframework.org/schema/p> "http://www.springframework.org/schema/p"; xmlns:c= <http://www.springframework.org/schema/c> "http://www.springframework.org/schema/c"; xmlns:util= <http://www.springframework.org/schema/util> "http://www.springframework.org/schema/util"; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd";> <description> Controls the generation of the unique identifiers for tickets. You most likely do not need to modify these. Though you may need to add the SAML ticket id generator. </description> <!-- ID Generators --> <bean id="ticketGrantingTicketUniqueIdGenerator" class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator" c:maxLength="50" c:suffix="${host.name}" /> <bean id="serviceTicketUniqueIdGenerator" class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator" c:maxLength="20" c:suffix="${host.name}" /> <bean id="loginTicketUniqueIdGenerator" class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator" c:maxLength="30" c:suffix="${host.name}" /> <bean id="proxy20TicketUniqueIdGenerator" class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator" c:maxLength="20" c:suffix="${host.name}" /> <!-- Needed for general SAML integration --> <bean id="samlServiceTicketUniqueIdGenerator" class="org.jasig.cas.support.saml.util.SamlCompliantUniqueTicketIdGenerator"> <constructor-arg index="0" value= <https://localhost:443> "https://localhost:443"; /> <!-- The below section gurantees SAML 2 for Google --> <property name="saml2compliant" value="true" /> </bean> <util:map id="uniqueIdGeneratorsMap"> <entry key="org.jasig.cas.authentication.principal.SimpleWebApplicationServiceImpl" value-ref="serviceTicketUniqueIdGenerator" /> <!-- Needed for general SAML integration --> <entry key="org.jasig.cas.support.saml.authentication.principal.SamlService" value-ref="samlServiceTicketUniqueIdGenerator" /> <!-- Needed for Google SAML integration --> <entry key="org.jasig.cas.support.saml.authentication.principal.GoogleAccountsService" value-ref="serviceTicketUniqueIdGenerator" /> </util:map> </beans> Our cas-servlet.xml file: <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to Jasig under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. Jasig licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at the following location: http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <beans xmlns= <http://www.springframework.org/schema/beans> "http://www.springframework.org/schema/beans"; xmlns:xsi= <http://www.w3.org/2001/XMLSchema-instance> "http://www.w3.org/2001/XMLSchema-instance"; xmlns:webflow= <http://www.springframework.org/schema/webflow-config> "http://www.springframework.org/schema/webflow-config"; xmlns:p= <http://www.springframework.org/schema/p> "http://www.springframework.org/schema/p"; xmlns:c= <http://www.springframework.org/schema/c> "http://www.springframework.org/schema/c"; xmlns:util= <http://www.springframework.org/schema/util> "http://www.springframework.org/schema/util"; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd http://www.springframework.org/schema/webflow-config http://www.springframework.org/schema/webflow-config/spring-webflow-config-2.3.xsd";> <import resource="spring-configuration/propertyFileConfigurer.xml"/> <!-- Theme Resolver --> <bean id="themeResolver" class="org.jasig.cas.services.web.ServiceThemeResolver" p:defaultThemeName="${cas.themeResolver.defaultThemeName}" p:argumentExtractors-ref="argumentExtractors" p:servicesManager-ref="servicesManager"> <property name="mobileBrowsers"> <util:map> <entry key=".*iPhone.*" value="iphone"/> <entry key=".*Android.*" value="iphone"/> <entry key=".*Safari.*Pre.*" value="iphone"/> <entry key=".*Nokia.*AppleWebKit.*" value="iphone"/> </util:map> </property> </bean> <!-- View Resolver --> <bean id="viewResolver" class="org.springframework.web.servlet.view.ResourceBundleViewResolver" p:order="0"> <property name="basenames"> <util:list> <value>${cas.viewResolver.basename}</value> <value>protocol_views</value> <!-- Needed for general SAML integration --> <value>saml_views</value> </util:list> </property> </bean> <!-- Locale Resolver --> <bean id="localeResolver" class="org.springframework.web.servlet.i18n.CookieLocaleResolver" p:defaultLocale="en" /> <bean id="localeChangeInterceptor" class="org.springframework.web.servlet.i18n.LocaleChangeInterceptor"/> <bean id="urlBasedViewResolver" class="org.springframework.web.servlet.view.UrlBasedViewResolver" p:viewClass="org.springframework.web.servlet.view.InternalResourceView" p:prefix="/WEB-INF/view/jsp/" p:suffix=".jsp" p:order="1"/> <bean id="errorHandlerResolver" class="org.jasig.cas.web.FlowExecutionExceptionResolver"/> <bean class="org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter"/> <bean id="handlerMappingC" class="org.springframework.web.servlet.handler.SimpleUrlHandlerMapping" p:alwaysUseFullPath="true"> <property name="mappings"> <util:properties> <prop key="/serviceValidate">serviceValidateController</prop> <prop key="/proxyValidate">proxyValidateController</prop> <prop key="/p3/serviceValidate">v3ServiceValidateController</prop> <prop key="/p3/proxyValidate">v3ProxyValidateController</prop> <prop key="/validate">legacyValidateController</prop> <prop key="/proxy">proxyController</prop> <prop key="/authorizationFailure.html">passThroughController</prop> <prop key="/status">healthCheckController</prop> <prop key="/statistics">statisticsController</prop> <!-- Necessary for SAML support --> <prop key="/samlValidate">samlValidateController</prop> </util:properties> </property> <!-- uncomment this to enable sending PageRequest events. <property name="interceptors"> <list> <ref bean="pageRequestHandlerInterceptorAdapter" /> </list> </property> --> </bean> <bean id="passThroughController" class="org.springframework.web.servlet.mvc.UrlFilenameViewController"/> <!-- login webflow configuration --> <bean id="loginFlowHandlerMapping" class="org.springframework.webflow.mvc.servlet.FlowHandlerMapping" p:flowRegistry-ref="loginFlowRegistry" p:order="2"> <property name="interceptors"> <ref local="localeChangeInterceptor" /> </property> </bean> <bean id="loginHandlerAdapter" class="org.jasig.cas.web.flow.SelectiveFlowHandlerAdapter" p:supportedFlowId="login" p:flowExecutor-ref="loginFlowExecutor" p:flowUrlHandler-ref="loginFlowUrlHandler" /> <bean id="loginFlowUrlHandler" class="org.jasig.cas.web.flow.CasDefaultFlowUrlHandler" /> <webflow:flow-executor id="loginFlowExecutor" flow-registry="loginFlowRegistry"> <webflow:flow-execution-attributes> <webflow:always-redirect-on-pause value="false" /> <webflow:redirect-in-same-state value="false" /> </webflow:flow-execution-attributes> <webflow:flow-execution-listeners> <webflow:listener ref="terminateWebSessionListener" /> </webflow:flow-execution-listeners> </webflow:flow-executor> <webflow:flow-registry id="loginFlowRegistry" flow-builder-services="builder"> <webflow:flow-location path="/WEB-INF/login-webflow.xml" id="login" /> </webflow:flow-registry> <!-- logout webflow configuration --> <bean id="logoutFlowHandlerMapping" class="org.springframework.webflow.mvc.servlet.FlowHandlerMapping" p:flowRegistry-ref="logoutFlowRegistry" p:order="3"> <property name="interceptors"> <ref local="localeChangeInterceptor" /> </property> </bean> <bean id="logoutHandlerAdapter" class="org.jasig.cas.web.flow.SelectiveFlowHandlerAdapter" p:supportedFlowId="logout" p:flowExecutor-ref="logoutFlowExecutor" p:flowUrlHandler-ref="logoutFlowUrlHandler" /> <bean id="logoutFlowUrlHandler" class="org.jasig.cas.web.flow.CasDefaultFlowUrlHandler" p:flowExecutionKeyParameter="RelayState" /> <webflow:flow-executor id="logoutFlowExecutor" flow-registry="logoutFlowRegistry"> <webflow:flow-execution-attributes> <webflow:always-redirect-on-pause value="false" /> <webflow:redirect-in-same-state value="false" /> </webflow:flow-execution-attributes> <webflow:flow-execution-listeners> <webflow:listener ref="terminateWebSessionListener" /> </webflow:flow-execution-listeners> </webflow:flow-executor> <webflow:flow-registry id="logoutFlowRegistry" flow-builder-services="builder"> <webflow:flow-location path="/WEB-INF/logout-webflow.xml" id="logout" /> </webflow:flow-registry> <webflow:flow-builder-services id="builder" view-factory-creator="viewFactoryCreator" expression-parser="expressionParser" /> <bean id="logoutConversionService" class="org.jasig.cas.web.flow.LogoutConversionService" /> <bean id="terminateWebSessionListener" class="org.jasig.cas.web.flow.TerminateWebSessionListener" /> <bean id="expressionParser" class="org.springframework.webflow.expression.spel.WebFlowSpringELExpressionParser" c:conversionService-ref="logoutConversionService"> <constructor-arg> <bean class="org.springframework.expression.spel.standard.SpelExpressionParser" /> </constructor-arg> </bean> <bean id="viewFactoryCreator" class="org.springframework.webflow.mvc.builder.MvcViewFactoryCreator"> <property name="viewResolvers"> <util:list> <ref local="viewResolver"/> </util:list> </property> </bean> <!-- CAS 2 Protocol service/proxy validation --> <bean id="abstractValidateController" class="org.jasig.cas.web.ServiceValidateController" abstract="true" p:centralAuthenticationService-ref="centralAuthenticationService" p:proxyHandler-ref="proxy20Handler" p:argumentExtractor-ref="casArgumentExtractor"/> <bean id="proxyValidateController" parent="abstractValidateController"/> <bean id="serviceValidateController" parent="abstractValidateController" p:validationSpecificationClass="org.jasig.cas.validation.Cas20WithoutProxyingValidationSpecification"/> <!-- CAS 3 Protocol service/proxy validation with attributes --> <bean id="v3AbstractValidateController" parent="abstractValidateController" abstract="true" p:successView="cas3ServiceSuccessView" p:failureView="cas3ServiceFailureView" /> <bean id="v3ProxyValidateController" parent="v3AbstractValidateController" /> <bean id="v3ServiceValidateController" parent="v3AbstractValidateController" p:validationSpecificationClass="org.jasig.cas.validation.Cas20WithoutProxyingValidationSpecification"/> <!-- CAS 1 legacy validation --> <bean id="legacyValidateController" parent="abstractValidateController" p:proxyHandler-ref="proxy10Handler" p:successView="cas1ServiceSuccessView" p:failureView="cas1ServiceFailureView" p:validationSpecificationClass="org.jasig.cas.validation.Cas10ProtocolValidationSpecification"/> <bean id="proxyController" class="org.jasig.cas.web.ProxyController" p:centralAuthenticationService-ref="centralAuthenticationService"/> <bean id="statisticsController" class="org.jasig.cas.web.StatisticsController" p:casTicketSuffix="${host.name}" c:ticketRegistry-ref="ticketRegistry" /> <bean id="logoutAction" class="org.jasig.cas.web.flow.LogoutAction" p:servicesManager-ref="servicesManager" p:followServiceRedirects="${cas.logout.followServiceRedirects:false}"/> <bean id="frontChannelLogoutAction" class="org.jasig.cas.web.flow.FrontChannelLogoutAction" c:logoutManager-ref="logoutManager"/> <bean id="healthCheckController" class="org.jasig.cas.web.HealthCheckController" p:healthCheckMonitor-ref="healthCheckMonitor"/> <bean id="initialFlowSetupAction" class="org.jasig.cas.web.flow.InitialFlowSetupAction" p:argumentExtractors-ref="argumentExtractors" p:warnCookieGenerator-ref="warnCookieGenerator" p:ticketGrantingTicketCookieGenerator-ref="ticketGrantingTicketCookieGenerator"/> <bean id="authenticationViaFormAction" class="org.jasig.cas.web.flow.AuthenticationViaFormAction" p:centralAuthenticationService-ref="centralAuthenticationService" p:warnCookieGenerator-ref="warnCookieGenerator" p:ticketRegistry-ref="ticketRegistry"/> <bean id="authenticationExceptionHandler" class="org.jasig.cas.web.flow.AuthenticationExceptionHandler" /> <bean id="generateServiceTicketAction" class="org.jasig.cas.web.flow.GenerateServiceTicketAction" p:centralAuthenticationService-ref="centralAuthenticationService"/> <bean id="sendTicketGrantingTicketAction" class="org.jasig.cas.web.flow.SendTicketGrantingTicketAction" p:centralAuthenticationService-ref="centralAuthenticationService" p:ticketGrantingTicketCookieGenerator-ref="ticketGrantingTicketCookieGenerator"/> <bean id="gatewayServicesManagementCheck" class="org.jasig.cas.web.flow.GatewayServicesManagementCheck" c:servicesManager-ref="servicesManager" /> <bean id="serviceAuthorizationCheck" class="org.jasig.cas.web.flow.ServiceAuthorizationCheck" c:servicesManager-ref="servicesManager" /> <bean id="generateLoginTicketAction" class="org.jasig.cas.web.flow.GenerateLoginTicketAction" p:ticketIdGenerator-ref="loginTicketUniqueIdGenerator"/> <bean id="messageInterpolator" class="org.jasig.cas.util.SpringAwareMessageMessageInterpolator"/> <bean id="credentialsValidator" class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean" p:messageInterpolator-ref="messageInterpolator"/> <bean id="ticketGrantingTicketCheckAction" class="org.jasig.cas.web.flow.TicketGrantingTicketCheckAction" c:registry-ref="ticketRegistry" /> <bean id="terminateSessionAction" class="org.jasig.cas.web.flow.TerminateSessionAction" c:cas-ref="centralAuthenticationService" c:tgtCookieGenerator-ref="ticketGrantingTicketCookieGenerator" c:warnCookieGenerator-ref="warnCookieGenerator"/> <!-- The following is used to configure SAML support. This is necessary for SAML support for Google --> <bean id="samlValidateController" class="org.jasig.cas.web.ServiceValidateController" p:validationSpecificationClass="org.jasig.cas.validation.Cas20WithoutProxyingValidationSpecification" p:centralAuthenticationService-ref="centralAuthenticationService" p:proxyHandler-ref="proxy20Handler" p:argumentExtractor-ref="samlArgumentExtractor" p:successView="casSamlServiceSuccessView" p:failureView="casSamlServiceFailureView"/> </beans> and our web.xml file: <?xml version="1.0" encoding="ISO-8859-1"?> <!-- Licensed to Jasig under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. Jasig licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at the following location: http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <web-app xmlns= <http://java.sun.com/xml/ns/j2ee> "http://java.sun.com/xml/ns/j2ee"; xmlns:xsi= <http://www.w3.org/2001/XMLSchema-instance> "http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation= <http://java.sun.com/xml/ns/j2eehttp:/java.sun.com/xml/ns/j2ee/web-app_2_4.xsd> "http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"; version="2.4"> <display-name>Central Authentication System (CAS) 4.0.0</display-name> <context-param> <param-name>contextConfigLocation</param-name> <param-value> /WEB-INF/spring-configuration/*.xml /WEB-INF/deployerConfigContext.xml </param-value> </context-param> <filter> <filter-name>CAS Client Info Logging Filter</filter-name> <filter-class>com.github.inspektr.common.web.ClientInfoThreadLocalFilter</filter-class> </filter> <filter-mapping> <filter-name>CAS Client Info Logging Filter</filter-name> <url-pattern>/*</url-pattern> </filter-mapping> <filter> <filter-name>springSecurityFilterChain</filter-name> <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class> </filter> <filter-mapping> <filter-name>springSecurityFilterChain</filter-name> <url-pattern>/status</url-pattern> </filter-mapping> <filter-mapping> <filter-name>springSecurityFilterChain</filter-name> <url-pattern>/statistics</url-pattern> </filter-mapping> <filter> <filter-name>characterEncodingFilter</filter-name> <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class> </filter> <filter-mapping> <filter-name>characterEncodingFilter</filter-name> <url-pattern>/*</url-pattern> </filter-mapping> <!-- - Loads the CAS ApplicationContext. - The deployer choice here is how to handle Throwables thrown by Spring's - ContextLoaderListener. The Spring ContextLoaderListener will throw an exception when the - application context cannot be loaded, say because the bean XML files are not valid XML or do not - refer to real classes and properties or because a bean configured via Spring throws an exception - at construction, property setting, or on an afterPropertiesSet() lifecycle method. - - If you'd like these errors to be fatal and prevent the CAS servlet context from loading at all, - use org.springframework.web.context.ContextLoaderListener. - - If you'd like these errors to result in all requests for CAS getting a "CAS is Unavailable" response, - use org.jasig.cas.web.init.SafeContextLoaderListener --> <listener> <listener-class> org.jasig.cas.web.init.SafeContextLoaderListener </listener-class> </listener> <!-- - This is the Spring dispatcher servlet which delegates all requests to the - Spring WebMVC controllers as configured in cas-servlet.xml. - - The choice made above about how to handle a broken ApplicationContext at - context initialization applies here as well, since this servlet is load-on-startup. - - If you'd like these errors to be fatal and prevent the CAS servlet from loading at all, - use org.springframework.web.servlet.DispatcherServlet. - - If you'd like these errors to result in all requests for CAS getting a "CAS is Unavailable" response, - use org.jasig.cas.web.init.SafeDispatcherServlet --> <servlet> <servlet-name>cas</servlet-name> <servlet-class> org.jasig.cas.web.init.SafeDispatcherServlet </servlet-class> <init-param> <param-name>publishContext</param-name> <param-value>false</param-value> </init-param> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/login</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/logout</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/validate</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/serviceValidate</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/p3/serviceValidate</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/proxy</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/proxyValidate</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/p3/proxyValidate</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/CentralAuthenticationService</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/status</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/statistics</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/authorizationFailure.html</url-pattern> </servlet-mapping> <!-- Needed for SAML integration --> <servlet-mapping> <servlet-name>cas</servlet-name> <url-pattern>/samlValidate</url-pattern> </servlet-mapping> <session-config> <!-- Default to 5 minute session timeouts --> <session-timeout>5</session-timeout> </session-config> <error-page> <error-code>401</error-code> <location>/authorizationFailure.html</location> </error-page> <error-page> <error-code>403</error-code> <location>/authorizationFailure.html</location> </error-page> <error-page> <error-code>404</error-code> <location>/</location> </error-page> <error-page> <error-code>500</error-code> <location>/WEB-INF/view/jsp/errors.jsp</location> </error-page> <error-page> <error-code>501</error-code> <location>/WEB-INF/view/jsp/errors.jsp</location> </error-page> <error-page> <error-code>503</error-code> <location>/WEB-INF/view/jsp/errors.jsp</location> </error-page> <welcome-file-list> <welcome-file>index.jsp</welcome-file> </welcome-file-list> </web-app> On 12/2/14 7:17 PM, Carl Waldbieser wrote: Dave, How many logins? We recently had a misconfugured cas client from a vendor almost fill /var. It was tens of thousands of logins. It would be nice if cas had some way to rate limit ST and login requests per user. Thanks, Carl On Dec 2, 2014 3:26 PM, "David A. Kovacic" <d...@case.edu> wrote: I'm not sure how or where you would mark this as a singleton instance - although if you go back to an actual Google web page multiple times from the same browser session you reuse the ST if that's what you mean. This actually looked like multiple logins from a single user over the span of about 30 minutes. Not sure if this was some poorly written webapp logging in several time or what. On 12/2/14 1:32 PM, Erik-Paul Dittmer wrote: Rapid heap memory consumption (which are not garbage collected) *can* be caused by unfinished Spring Webflow flow sessions; this is something we have observed. However, when looking at your memory dump, the majority of the instances (and size) is being claimed by the GoogleAccountService. Perhaps this is not marked as a singleton instance? On Tue, Dec 2, 2014 at 6:38 PM, David A. Kovacic <d...@case.edu> wrote: All, Yesterday evening one of our CAS 4.0.0 servers went from under a GB of heap usage to 3GB in a matter of about 10 minutes. The end result was that again the SSO service died (one server with a heap memory OoM error and the other trying to replicate the ehcache to the dead server. This was definitely not a memory leak issue as the servers had been restarted only earlier that morning, so they had only been up for about 17 hours or so. Out system monitors also indicated that the memory usage rather suddenly skyrocketed (over the course of about 20 minutes) so we suspect that the memory consumption is a symptom of some other issue. We have a heap dump but I am having a bit of trouble trying to analyze it with jvisualvm as I have never used the tool before. If I am interpreting the dump correctly, it appears that tickets only play a very small part of the overall memory usage (see screen shot). Has anyone heard or experienced anything like what we are seeing? This is becoming increasingly frustrating as every time we think we have the issues resolved and turn our attention elsewhere one server or the other crashes and takes the service down with it. Dave -- You are currently subscribed to cas-user@lists.jasig.org as: epditt...@digitalmisfits.com To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- Erik-Paul Dittmer T: +31 (0) 64 761 87 57 Visit us at http://www.digitalmisfits.com - - - - - - - - - - - - - - - - - - - - - - - - - - Digital Misfits does not accept any liability for any errors, omissions, delays of receipt or viruses in the contents of this message which arise as a result of e-mail transmission. -- You are currently subscribed to cas-user@lists.jasig.org as: d...@case.edu To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to cas-user@lists.jasig.org as: cwaldbie...@gmail.com To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to cas-user@lists.jasig.org as: d...@case.edu To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to cas-user@lists.jasig.org as: cfern...@sju.edu To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user -- You are currently subscribed to cas-user@lists.jasig.org as: arch...@mail-archive.com To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
