I am wondering if this is related to https://github.com/apereo/cas/pull/5652. We have been discussing this issue because myself and some others have experienced a slowdown due to the merged pull request but the fix provided by the pull request is related to fixing out-of-memory issues with the embedded Tomcat.
On Thursday, November 16, 2023 at 7:26:03 AM UTC-5 Mohamed Amdouni wrote: > These exceptions will not help you. They are consequences. Check instead > the server startup log files. Your memory settings as sayed before. > > > > Le jeu. 16 nov. 2023 à 13:12, Sathish Sekar <[email protected]> a > écrit : > >> Hi Team, >> >> Additionally I can find exceptions for java heap memory error in prod. >> Kindly do needful please >> >> AsyncLogger error handling event seq=11325854, value=' >> Logger=com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl >> Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Java heap space': >> org.apache.logging.log4j.core.appender.AppenderLoggingException: >> java.lang.NoClassDefFoundError: Could not initialize class >> org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil >> >> AsyncLogger error handling event seq=11325852, value=' >> Logger=com.hazelcast.internal.partition.operation.FinalizePromotionOperation >> Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Java heap space': >> org.apache.logging.log4j.core.appender.AppenderLoggingException: >> java.lang.NoClassDefFoundError: Could not initialize class >> org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil >> >> org.apache.logging.log4j.core.appender.AppenderLoggingException: >> java.lang.OutOfMemoryError: Java heap space at >> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender( >> AppenderControl.java:165) at >> org.apache.logging.log4j.core.config.AppenderControl.callAppender0( >> AppenderControl.java:134) at >> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion >> (AppenderControl.java:125) at >> org.apache.logging.log4j.core.config.AppenderControl.callAppender( >> AppenderControl.java:89) at >> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders( >> LoggerConfig.java:675) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders( >> AsyncLoggerConfig.java:149) at >> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent( >> LoggerConfig.java:633) at >> org.apache.logging.log4j.core.config.LoggerConfig.log( >> LoggerConfig.java:616) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfig.log( >> AsyncLoggerConfig.java:143) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread >> (AsyncLoggerConfig.java:191) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >> (AsyncLoggerConfigDisruptor.java:112) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >> (AsyncLoggerConfigDisruptor.java:98) at >> com.lmax.disruptor.BatchEventProcessor.processEvents( >> BatchEventProcessor.java:168) at >> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) >> at java.base/java.lang.Thread.run(Thread.java:829) >> >> AsyncLogger error handling event seq=11325849, value=' >> Logger=com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl >> Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Failed to process: >> com.hazelcast.internal.partition.operation.FinalizePromotionOperation{ >> serviceName='null', identityHash=1381425466, partitionId=44, >> replicaIndex=0, callId=0, invocationTime=-1 (1969-12-31 15:59:59.999), >> waitTimeout=-1, callTimeout=9223372036854775807, >> tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0} >> on: hz.10.34.196.43.partition-operation.thread-0': >> org.apache.logging.log4j.core.appender.AppenderLoggingException: >> java.lang.OutOfMemoryError: Java heap space >> >> Log4j2-TF-15-AsyncLoggerConfig-11 ERROR An exception occurred processing >> Appender casFile >> org.apache.logging.log4j.core.appender.AppenderLoggingException: >> java.lang.OutOfMemoryError: Java heap space at >> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender( >> AppenderControl.java:165) at >> org.apache.logging.log4j.core.config.AppenderControl.callAppender0( >> AppenderControl.java:134) at >> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion >> (AppenderControl.java:125) at >> org.apache.logging.log4j.core.config.AppenderControl.callAppender( >> AppenderControl.java:89) at >> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders( >> LoggerConfig.java:675) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders( >> AsyncLoggerConfig.java:149) at >> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent( >> LoggerConfig.java:633) at >> org.apache.logging.log4j.core.config.LoggerConfig.log( >> LoggerConfig.java:616) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfig.log( >> AsyncLoggerConfig.java:143) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread >> (AsyncLoggerConfig.java:191) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >> (AsyncLoggerConfigDisruptor.java:112) at >> org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent >> (AsyncLoggerConfigDisruptor.java:98) at >> com.lmax.disruptor.BatchEventProcessor.processEvents( >> BatchEventProcessor.java:168) at >> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) >> at java.base/java.lang.Thread.run(Thread.java:829) >> >> Exception in thread "HTTP-Dispatcher" java.lang.OutOfMemoryError: Java heap >> space at java.base/java.util.HashMap$KeySet.iterator(HashMap.java:913) at >> java.base/java.util.HashSet.iterator(HashSet.java:173) at >> java.base/sun.nio.ch.Util$2.iterator(Util.java:352) >> On Thursday, November 16, 2023 at 4:42:37 PM UTC+5:30 Sathish Sekar wrote: >> >>> Hi Team, >>> >>> Kindly help on the below exception for memory issue. >>> >>> java.lang.OutOfMemoryError: Java heap space at java.lang.Integer.valueOf >>> (Integer.java:1059) ~[?:?] at sun.nio.ch.EPollSelectorImpl.processEvents >>> (EPollSelectorImpl.java:194) ~[?:?] at >>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:137) ~[?:?] >>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) ~[?:?] >>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136) ~[?:?] at >>> com.hazelcast.internal.networking.nio.NioThread.selectLoop( >>> NioThread.java:292) ~[hazelcast-5.1.7.jar:5.1.7] at >>> com.hazelcast.internal.networking.nio.NioThread.executeRun( >>> NioThread.java:249) ~[hazelcast-5.1.7.jar:5.1.7] at >>> com.hazelcast.internal.util.executor.HazelcastManagedThread.run( >>> HazelcastManagedThread.java:102) ~[hazelcast-5.1.7.jar:5.1.7] >>> >>> On Thursday, November 16, 2023 at 1:01:30 PM UTC+5:30 Mohamed Amdouni >>> wrote: >>> >>>> Hi, >>>> Please post the stack trace >>>> Did you configurer the Jvm to generate a heap dump when out of memory >>>> exception is encouraged ? >>>> Is it about meta space ? >>>> >>>> Best regards >>>> >>>> Le jeu. 16 nov. 2023 à 05:35, Sathish Sekar <[email protected]> a >>>> écrit : >>>> >>>>> Hi Team, >>>>> >>>>> I have upgraded to CAS 6.3.7.4 to 6.6.9 recently and I'm facing java >>>>> heap memory error. In CAS 6.3.7.4 I didn't face such issues in prod >>>>> environment. When I upgraded to CAS 6.6.9 I had faced this issue thrice >>>>> and >>>>> I have posted this in community and as per their feedback and as per the >>>>> CAS documentation I have upgraded memory maximum 2GB as earlier I have >>>>> maximum of 512mb of memory in prod. After upgrade application is running >>>>> for 10 days and suddenly it stopped showing java heap memory error in >>>>> catalina.out.log. Kindly suggest any other possible solutions for heap >>>>> memory errors so that I can fix in prod environment. >>>>> >>>>> *Note*: >>>>> Since I have vulnerabilities in CAS 6.6.9 I have excluded following >>>>> jars and included non vulnerable version which is highlighted in bold. >>>>> >>>>> spring-security-config-5.7.3.jar >>>>> *spring-security-config-5.7.10.jar* >>>>> spring-security-web-5.7.3.jar >>>>> *spring-security-web-5.7.10.jar* >>>>> spring-web-5.3.22.jar >>>>> *spring-web-5.3.30.jar* >>>>> hazelcast-5.1.3.jar >>>>> *hazelcast-5.1.7.jar* >>>>> hazelcast-sql-5.1.3.jar >>>>> *hazelcast-sql-5.1.7.jar* >>>>> spring-boot-actuator-autoconfigure-2.7.3.jar >>>>> *spring-boot-actuator-autoconfigure-2.7.11.jar* >>>>> json-20160810.jar >>>>> *json-20230227.jar* >>>>> json-smart-2.4.8.jar >>>>> *json-smart-2.4.9.jar* >>>>> spring-webmvc-5.3.22.jar >>>>> *spring-webmvc-5.3.30.jar* >>>>> spring-boot-autoconfigure-2.7.3.jar >>>>> *spring-boot-autoconfigure-2.7.11.jar* >>>>> hjson-3.0.0.jar >>>>> *hjson-3.0.1.jar* >>>>> guava-30.1.1-jre.jar >>>>> *guava-32.0.1-jre.jar* >>>>> tomcat-catalina-9.0.75.jar >>>>> *tomcat-catalina-9.0.76.jar* >>>>> tomcat-embed-core-9.0.75.jar >>>>> *tomcat-embed-core-9.0.76.jar* >>>>> jose4j-0.8.0.jar >>>>> *jose4j-0.9.3.jar* >>>>> jersey-common-2.25.1.jar >>>>> *jersey-common-2.34.jar* >>>>> bcprov-jdk18on-1.71.jar >>>>> *bcprov-jdk18on-1.74.jar* >>>>> spring-expression-5.3.22.jar >>>>> *spring-expression-5.3.30.jar* >>>>> spring-security-crypto-5.7.3.jar >>>>> *spring-security-crypto-5.7.10.jar* >>>>> woodstox-core-6.3.1.jar >>>>> *woodstox-core-6.4.0.jar* >>>>> jackson-databind-2.13.4.jar >>>>> *jackson-databind-2.13.5.jar* >>>>> jackson-core-2.13.4.jar >>>>> *jackson-core-2.13.5.jar* >>>>> >>>>> -- >>>>> - Website: https://apereo.github.io/cas >>>>> - Gitter Chatroom: https://gitter.im/apereo/cas >>>>> - List Guidelines: https://goo.gl/1VRrw7 >>>>> - Contributions: https://goo.gl/mh7qDG >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "CAS Community" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/a/apereo.org/d/msgid/cas-user/0880771f-aff5-4009-a892-0bdcdacc4ecbn%40apereo.org >>>>> >>>>> <https://groups.google.com/a/apereo.org/d/msgid/cas-user/0880771f-aff5-4009-a892-0bdcdacc4ecbn%40apereo.org?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/b4ea5e9e-3cb6-401e-951c-4e2bd39e7ea1n%40apereo.org.
