The issue seems to be in the EHCache. CAS itself does not issue any
errors but just prior to the OoM error we see a couple of:
2014-11-13 21:54:02,820 ERROR
[net.sf.ehcache.distribution.RMISynchronousCacheReplicator] - <Exception
on replication of putNotification. GC overhead limit exceeded.
Continuing...>
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3658)
at java.lang.StringBuffer.toString(StringBuffer.java:671)
at
java.net.URLStreamHandler.toExternalForm(URLStreamHandler.java:507)
at java.net.URL.toExternalForm(URL.java:922)
at sun.rmi.server.LoaderHandler.urlsToPath(LoaderHandler.java:799)
at
sun.rmi.server.LoaderHandler.getClassAnnotation(LoaderHandler.java:264)
at
java.rmi.server.RMIClassLoader$2.getClassAnnotation(RMIClassLoader.java:657)
at
java.rmi.server.RMIClassLoader.getClassAnnotation(RMIClassLoader.java:381)
at
sun.rmi.server.MarshalOutputStream.annotateClass(MarshalOutputStream.java:93)
at
java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1290)
at
java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:294)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:153)
at net.sf.ehcache.distribution.RMICachePeer_Stub.put(Unknown Source)
at
net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:149)
at
net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementPut(RMISynchronousCacheReplicator.java:132)
at
net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementPut(RegisteredEventListeners.java:180)
at
net.sf.ehcache.event.RegisteredEventListeners.notifyElementPut(RegisteredEventListeners.java:158)
at net.sf.ehcache.Cache.notifyPutInternalListeners(Cache.java:1532)
at net.sf.ehcache.Cache.putInternal(Cache.java:1502)
at net.sf.ehcache.Cache.put(Cache.java:1427)
at net.sf.ehcache.Cache.put(Cache.java:1392)
at
org.jasig.cas.ticket.registry.EhCacheTicketRegistry.addTicket(EhCacheTicketRegistry.java:81)
at
org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:315)
at
org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:344)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
Immediately after that we see
2014-11-13 21:54:08,169 ERROR
[net.sf.ehcache.store.disk.DiskStorageFactory] - <Disk Write of
ST-230861-YA2VSPgftxLvviu7U75v-ssop-cfd-1.cwru.edu failed: >
java.lang.OutOfMemoryError: GC overhead limit exceeded
and then shortly thereafter we start to see:
2014-11-13 22:09:36,424 ERROR
[net.sf.ehcache.store.disk.DiskStorageFactory] - <Disk Write of
ST-231391-DOCz33CrEBWikDcApg5E-ssop-cfd-1.cwru.edu failed: >
java.lang.OutOfMemoryError: Java heap space
2014-11-13 22:09:37,026 ERROR
[net.sf.ehcache.store.disk.DiskStorageFactory] - <Disk Write of
ST-231395-Ve3QcSjOdhw2n5TfIdm5-ssop-cfd-1.cwru.edu failed: >
java.lang.OutOfMemoryError: Java heap space
2014-11-13 22:09:37,026 ERROR
[net.sf.ehcache.store.disk.DiskStorageFactory] - <Disk Write of
ST-231394-ZJKtdcYZ1bgcj7Ulx53H-ssop-cfd-1.cwru.edu failed: >
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-443-exec-6640" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "http-bio-80-exec-1" java.lang.OutOfMemoryError:
Java heap space
2014-11-13 22:09:40,316 ERROR
[net.sf.ehcache.store.disk.DiskStorageFactory] - <Disk Write of
TGT-166430-GqTNDEgnTAUdFfJfKxIpara5zNR6K2lHtVebEAd7BcR2AscMyA-ssop-cfd-1.cwru.edu
failed: >
java.lang.OutOfMemoryError: Java heap space
2014-11-13 22:09:40,316 ERROR
[net.sf.ehcache.store.disk.DiskStorageFactory] - <Disk Write of
TGT-167049-6TBqruoPAd2EjwameWKkppF3evcvyiIcroTojple7ScfRYmAXZ-ssop-cfd-1.cwru.edu
failed: >
java.lang.OutOfMemoryError: Java heap space
2014-11-13 22:09:40,317 ERROR
[net.sf.ehcache.store.disk.DiskStorageFactory] - <Disk Write of
TGT-167049-6TBqruoPAd2EjwameWKkppF3evcvyiIcroTojple7ScfRYmAXZ-ssop-cfd-1.cwru.edu
failed: >
java.lang.OutOfMemoryError: Java heap space
which repeats until the service is stopped and started. Typically that
locks up the other server in an attempt to replicate teh tickets created
there.
On 11/14/14 9:46 AM, Marvin Addison wrote:
>
> For the the second time both of our SSO servers running under
> Tomcat ran
> out of heap memory last night.
>
>
> Did CAS emit any errors to the application log file, cas.log, prior to
> OOM? Please post anything you have. You can configure the JVM to
> perform a heap dump prior to exiting, which you should set up
> immediately so you have some additional information if/when it happens
> next time. The follow system properties should do the trick:
>
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/path/to/jvm-dump.hprof"
>
> You might also consider taking periodic heap dumps using the jmap tool
> and analyzing offline using some analysis tool. We use the YourKit
> Java profile for this kind of thing, fwiw.
>
> Hope that helps,
> M
>
> --
> You are currently subscribed to cas-user@lists.jasig.org as: d...@case.edu
> To unsubscribe, change settings or access archives, see
> http://www.ja-sig.org/wiki/display/JSG/cas-user
--
You are currently subscribed to cas-user@lists.jasig.org as:
arch...@mail-archive.com
To unsubscribe, change settings or access archives, see
http://www.ja-sig.org/wiki/display/JSG/cas-user
