Hello, Are you hazelcast registry? if so, we encountered a similar problem in our prod environment. Especially after we converted one of our popular app from cas to Oauth. Here is what our config that made out cas stable, without restarting on a schedule
cas.ticket.registry.hazelcast.cluster.core.eviction-policy=LRU cas.ticket.registry.hazelcast.cluster.core.max-size-policy=USED_HEAP_PERCENTAGE cas.ticket.registry.hazelcast.cluster.core.max-size=50 VM has 9G of ram OpenJDK 11 -Xms6144m -Xmx6144m Hopefully this helps Winson On Thursday, November 16, 2023 at 9:50:01 a.m. UTC-6 Doug C wrote: > I am wondering if this is related to > https://github.com/apereo/cas/pull/5652. We have been discussing this > issue because myself and some others have experienced a slowdown due to the > merged pull request but the fix provided by the pull request is related to > fixing out-of-memory issues with the embedded Tomcat. > > On Thursday, November 16, 2023 at 7:26:03 AM UTC-5 Mohamed Amdouni wrote: > >> These exceptions will not help you. They are consequences. Check instead >> the server startup log files. Your memory settings as sayed before. >> >> >> >> Le jeu. 16 nov. 2023 à 13:12, Sathish Sekar <[email protected]> a >> écrit : >> >>> Hi Team, >>> >>> Additionally I can find exceptions for java heap memory error in prod. >>> Kindly do needful please >>> >>> AsyncLogger error handling event seq=11325854, value=' >>> Logger=com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl >>> Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Java heap space': >>> org.apache.logging.log4j.core.appender.AppenderLoggingException: >>> java.lang.NoClassDefFoundError: Could not initialize class >>> org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil >>> >>> AsyncLogger error handling event seq=11325852, value=' >>> Logger=com.hazelcast.internal.partition.operation.FinalizePromotionOperation >>> >>> Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Java heap space': >>> org.apache.logging.log4j.core.appender.AppenderLoggingException: >>> java.lang.NoClassDefFoundError: Could not initialize class >>> org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil >>> >>> org.apache.logging.log4j.core.appender.AppenderLoggingException: >>> java.lang.OutOfMemoryError: Java heap space at >>> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender( >>> AppenderControl.java:165) at >>> org.apache.logging.log4j.core.config.AppenderControl.callAppender0( >>> AppenderControl.java:134) at >>> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion >>> (AppenderControl.java:125) at >>> org.apache.logging.log4j.core.config.AppenderControl.callAppender( >>> AppenderControl.java:89) at >>> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders( >>> LoggerConfig.java:675) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders( >>> AsyncLoggerConfig.java:149) at >>> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent( >>> LoggerConfig.java:633) at >>> org.apache.logging.log4j.core.config.LoggerConfig.log( >>> LoggerConfig.java:616) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfig.log( >>> AsyncLoggerConfig.java:143) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread >>> (AsyncLoggerConfig.java:191) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >>> (AsyncLoggerConfigDisruptor.java:112) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >>> (AsyncLoggerConfigDisruptor.java:98) at >>> com.lmax.disruptor.BatchEventProcessor.processEvents( >>> BatchEventProcessor.java:168) at >>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) >>> at java.base/java.lang.Thread.run(Thread.java:829) >>> >>> AsyncLogger error handling event seq=11325849, value=' >>> Logger=com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl >>> Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Failed to >>> process: >>> com.hazelcast.internal.partition.operation.FinalizePromotionOperation{ >>> serviceName='null', identityHash=1381425466, partitionId=44, >>> replicaIndex=0, callId=0, invocationTime=-1 (1969-12-31 15:59:59.999), >>> waitTimeout=-1, callTimeout=9223372036854775807, >>> tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0} >>> on: hz.10.34.196.43.partition-operation.thread-0': >>> org.apache.logging.log4j.core.appender.AppenderLoggingException: >>> java.lang.OutOfMemoryError: Java heap space >>> >>> Log4j2-TF-15-AsyncLoggerConfig-11 ERROR An exception occurred processing >>> Appender casFile >>> org.apache.logging.log4j.core.appender.AppenderLoggingException: >>> java.lang.OutOfMemoryError: Java heap space at >>> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender( >>> AppenderControl.java:165) at >>> org.apache.logging.log4j.core.config.AppenderControl.callAppender0( >>> AppenderControl.java:134) at >>> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion >>> (AppenderControl.java:125) at >>> org.apache.logging.log4j.core.config.AppenderControl.callAppender( >>> AppenderControl.java:89) at >>> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders( >>> LoggerConfig.java:675) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders( >>> AsyncLoggerConfig.java:149) at >>> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent( >>> LoggerConfig.java:633) at >>> org.apache.logging.log4j.core.config.LoggerConfig.log( >>> LoggerConfig.java:616) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfig.log( >>> AsyncLoggerConfig.java:143) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread >>> (AsyncLoggerConfig.java:191) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >>> (AsyncLoggerConfigDisruptor.java:112) at >>> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >>> (AsyncLoggerConfigDisruptor.java:98) at >>> com.lmax.disruptor.BatchEventProcessor.processEvents( >>> BatchEventProcessor.java:168) at >>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) >>> at java.base/java.lang.Thread.run(Thread.java:829) >>> >>> Exception in thread "HTTP-Dispatcher" java.lang.OutOfMemoryError: Java heap >>> space at java.base/java.util.HashMap$KeySet.iterator(HashMap.java:913) >>> at java.base/java.util.HashSet.iterator(HashSet.java:173) at >>> java.base/sun.nio.ch.Util$2.iterator(Util.java:352) >>> On Thursday, November 16, 2023 at 4:42:37 PM UTC+5:30 Sathish Sekar >>> wrote: >>> >>>> Hi Team, >>>> >>>> Kindly help on the below exception for memory issue. >>>> >>>> java.lang.OutOfMemoryError: Java heap space at >>>> java.lang.Integer.valueOf(Integer.java:1059) ~[?:?] at >>>> sun.nio.ch.EPollSelectorImpl.processEvents(EPollSelectorImpl.java:194) >>>> ~[?:?] at sun.nio.ch.EPollSelectorImpl.doSelect( >>>> EPollSelectorImpl.java:137) ~[?:?] at >>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) ~[?:?] >>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136) ~[?:?] at >>>> com.hazelcast.internal.networking.nio.NioThread.selectLoop( >>>> NioThread.java:292) ~[hazelcast-5.1.7.jar:5.1.7] at >>>> com.hazelcast.internal.networking.nio.NioThread.executeRun( >>>> NioThread.java:249) ~[hazelcast-5.1.7.jar:5.1.7] at >>>> com.hazelcast.internal.util.executor.HazelcastManagedThread.run( >>>> HazelcastManagedThread.java:102) ~[hazelcast-5.1.7.jar:5.1.7] >>>> >>>> On Thursday, November 16, 2023 at 1:01:30 PM UTC+5:30 Mohamed Amdouni >>>> wrote: >>>> >>>>> Hi, >>>>> Please post the stack trace >>>>> Did you configurer the Jvm to generate a heap dump when out of memory >>>>> exception is encouraged ? >>>>> Is it about meta space ? >>>>> >>>>> Best regards >>>>> >>>>> Le jeu. 16 nov. 2023 à 05:35, Sathish Sekar <[email protected]> a >>>>> écrit : >>>>> >>>>>> Hi Team, >>>>>> >>>>>> I have upgraded to CAS 6.3.7.4 to 6.6.9 recently and I'm facing java >>>>>> heap memory error. In CAS 6.3.7.4 I didn't face such issues in prod >>>>>> environment. When I upgraded to CAS 6.6.9 I had faced this issue thrice >>>>>> and >>>>>> I have posted this in community and as per their feedback and as per the >>>>>> CAS documentation I have upgraded memory maximum 2GB as earlier I have >>>>>> maximum of 512mb of memory in prod. After upgrade application is running >>>>>> for 10 days and suddenly it stopped showing java heap memory error in >>>>>> catalina.out.log. Kindly suggest any other possible solutions for heap >>>>>> memory errors so that I can fix in prod environment. >>>>>> >>>>>> *Note*: >>>>>> Since I have vulnerabilities in CAS 6.6.9 I have excluded following >>>>>> jars and included non vulnerable version which is highlighted in bold. >>>>>> >>>>>> spring-security-config-5.7.3.jar >>>>>> *spring-security-config-5.7.10.jar* >>>>>> spring-security-web-5.7.3.jar >>>>>> *spring-security-web-5.7.10.jar* >>>>>> spring-web-5.3.22.jar >>>>>> *spring-web-5.3.30.jar* >>>>>> hazelcast-5.1.3.jar >>>>>> *hazelcast-5.1.7.jar* >>>>>> hazelcast-sql-5.1.3.jar >>>>>> *hazelcast-sql-5.1.7.jar* >>>>>> spring-boot-actuator-autoconfigure-2.7.3.jar >>>>>> *spring-boot-actuator-autoconfigure-2.7.11.jar* >>>>>> json-20160810.jar >>>>>> *json-20230227.jar* >>>>>> json-smart-2.4.8.jar >>>>>> *json-smart-2.4.9.jar* >>>>>> spring-webmvc-5.3.22.jar >>>>>> *spring-webmvc-5.3.30.jar* >>>>>> spring-boot-autoconfigure-2.7.3.jar >>>>>> *spring-boot-autoconfigure-2.7.11.jar* >>>>>> hjson-3.0.0.jar >>>>>> *hjson-3.0.1.jar* >>>>>> guava-30.1.1-jre.jar >>>>>> *guava-32.0.1-jre.jar* >>>>>> tomcat-catalina-9.0.75.jar >>>>>> *tomcat-catalina-9.0.76.jar* >>>>>> tomcat-embed-core-9.0.75.jar >>>>>> *tomcat-embed-core-9.0.76.jar* >>>>>> jose4j-0.8.0.jar >>>>>> *jose4j-0.9.3.jar* >>>>>> jersey-common-2.25.1.jar >>>>>> *jersey-common-2.34.jar* >>>>>> bcprov-jdk18on-1.71.jar >>>>>> *bcprov-jdk18on-1.74.jar* >>>>>> spring-expression-5.3.22.jar >>>>>> *spring-expression-5.3.30.jar* >>>>>> spring-security-crypto-5.7.3.jar >>>>>> *spring-security-crypto-5.7.10.jar* >>>>>> woodstox-core-6.3.1.jar >>>>>> *woodstox-core-6.4.0.jar* >>>>>> jackson-databind-2.13.4.jar >>>>>> *jackson-databind-2.13.5.jar* >>>>>> jackson-core-2.13.4.jar >>>>>> *jackson-core-2.13.5.jar* >>>>>> >>>>>> -- >>>>>> - Website: https://apereo.github.io/cas >>>>>> - Gitter Chatroom: https://gitter.im/apereo/cas >>>>>> - List Guidelines: https://goo.gl/1VRrw7 >>>>>> - Contributions: https://goo.gl/mh7qDG >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "CAS Community" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/a/apereo.org/d/msgid/cas-user/0880771f-aff5-4009-a892-0bdcdacc4ecbn%40apereo.org >>>>>> >>>>>> <https://groups.google.com/a/apereo.org/d/msgid/cas-user/0880771f-aff5-4009-a892-0bdcdacc4ecbn%40apereo.org?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/af95e566-8474-420d-9821-bd1c8b196498n%40apereo.org.
