Re: Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 5/30/2011 12:24 PM, André Warnier wrote: > Note also that running out of Heap space does not necessarily mean that > your classes have leaks. It can also mean that they are just using > memory to a point where your allocated Heap space is simply not sufficient. +1 > Looking at the stack trace below, I find it unlikely that the code > itself in > org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate > , by itself, would use up the Heap. However, if by the time it is > called, the Heap is already 99% full, it just might. +1 OOMEs can occur at any time. It's almost never a problem in the code shown in the stack trace. Most OOMEs I have seen do not even contain stack traces. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3lSUwACgkQ9CaO5/Lv0PDIpwCfTwdqlybK7yao6/5DljIbRVr6 XSYAoIs02TaPShxzXgfAyJWobSlm4s82 =/u2C -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space
By the way, in addition to the details below, you may want to also indicate with what Heap size you are running Tomcat. I suppose that the IBM Heap analyser you are using would tell you that. Otherwise, if you are running Tomcat as a Windows Service, use the GUI application (tomcat7w.exe) to examine the Java JVM options tab. Note also that running out of Heap space does not necessarily mean that your classes have leaks. It can also mean that they are just using memory to a point where your allocated Heap space is simply not sufficient. Starting Tomcat with a Heap of 64M and starting 1000 request threads simultaneously would do it nicely also. Looking at the stack trace below, I find it unlikely that the code itself in org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate , by itself, would use up the Heap. However, if by the time it is called, the Heap is already 99% full, it just might. André Warnier wrote: Well maybe then you should have posted that stack trace and the comments below to the list, and maybe someone would then have taken a closer look. I apologise (a little bit) for my original answer, but you must admit that with the information you posted before, it was hard to make the difference between that, and someone posting : "Tomcat doesn't work. Can you help ?" sunil.sheva...@wipro.com wrote: Hi Andre, I am not kidding. My application has only 8 to 10 classes. I checked the heap using IBM heap analyser and my objects are not leaking memory. I am guessing that this is a bug in tomcat 7. Let me know in case you can make any sense out of the trace. === Exception in thread ""http-bio-8080"-exec-12" java.lang.OutOfMemoryError: Java heap space at java.util.LinkedHashMap.createEntry(Unknown Source) at java.util.LinkedHashMap.addEntry(Unknown Source) at java.util.HashMap.put(Unknown Source) at sun.util.resources.OpenListResourceBundle.loadLookup(Unknown Source) at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(Unknown Source) at sun.util.resources.OpenListResourceBundle.handleGetObject(Unknown Source) at sun.util.resources.TimeZoneNamesBundle.handleGetObject(Unknown Source) at java.util.ResourceBundle.getObject(Unknown Source) at java.util.ResourceBundle.getObject(Unknown Source) at java.util.ResourceBundle.getStringArray(Unknown Source) at sun.util.TimeZoneNameUtility.retrieveDisplayNames(Unknown Source) at sun.util.TimeZoneNameUtility.retrieveDisplayNames(Unknown Source) at java.util.TimeZone.getDisplayNames(Unknown Source) at java.util.TimeZone.getDisplayName(Unknown Source) at java.text.SimpleDateFormat.subFormat(Unknown Source) at java.text.SimpleDateFormat.format(Unknown Source) at java.text.SimpleDateFormat.format(Unknown Source) at java.text.DateFormat.format(Unknown Source) at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHttpDateFormat.java:115) at org.apache.coyote.http11.AbstractHttp11Processor.prepareResponse(AbstractHttp11Processor.java:926) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:739) at org.apache.coyote.Response.action(Response.java:170) at org.apache.coyote.Response.sendHeaders(Response.java:350) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:308) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:275) at org.apache.catalina.connector.Response.finishResponse(Response.java:501) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:243) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:188) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:166) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:288) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space at java.lang.reflect.Array.newArray(Native Method) at java.lang.reflect.Array.newInstance(Unknown Source) at java.util.AbstractCollection.toArray(Unknown Source) at org.apache.catalina.session.ManagerBase.findSessions(ManagerBase.java:705) at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:527) at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:518) at org.apache.
RE: Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space
> From: André Warnier [mailto:a...@ice-sa.com] > Subject: Re: Exception in thread ""http-bio-8080"-exec-9" > java.lang.OutOfMemoryError: Java heap space > I apologise (a little bit) for my original answer You shouldn't apologize; the OP has still not provided any useful information (just denial of responsibility), and has violated mailing list protocol by replying directly to you rather than to the list. He's nil-for-two at this point; how many wickets do we give him? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space
Well maybe then you should have posted that stack trace and the comments below to the list, and maybe someone would then have taken a closer look. I apologise (a little bit) for my original answer, but you must admit that with the information you posted before, it was hard to make the difference between that, and someone posting : "Tomcat doesn't work. Can you help ?" sunil.sheva...@wipro.com wrote: Hi Andre, I am not kidding. My application has only 8 to 10 classes. I checked the heap using IBM heap analyser and my objects are not leaking memory. I am guessing that this is a bug in tomcat 7. Let me know in case you can make any sense out of the trace. === Exception in thread ""http-bio-8080"-exec-12" java.lang.OutOfMemoryError: Java heap space at java.util.LinkedHashMap.createEntry(Unknown Source) at java.util.LinkedHashMap.addEntry(Unknown Source) at java.util.HashMap.put(Unknown Source) at sun.util.resources.OpenListResourceBundle.loadLookup(Unknown Source) at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(Unknown Source) at sun.util.resources.OpenListResourceBundle.handleGetObject(Unknown Source) at sun.util.resources.TimeZoneNamesBundle.handleGetObject(Unknown Source) at java.util.ResourceBundle.getObject(Unknown Source) at java.util.ResourceBundle.getObject(Unknown Source) at java.util.ResourceBundle.getStringArray(Unknown Source) at sun.util.TimeZoneNameUtility.retrieveDisplayNames(Unknown Source) at sun.util.TimeZoneNameUtility.retrieveDisplayNames(Unknown Source) at java.util.TimeZone.getDisplayNames(Unknown Source) at java.util.TimeZone.getDisplayName(Unknown Source) at java.text.SimpleDateFormat.subFormat(Unknown Source) at java.text.SimpleDateFormat.format(Unknown Source) at java.text.SimpleDateFormat.format(Unknown Source) at java.text.DateFormat.format(Unknown Source) at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHttpDateFormat.java:115) at org.apache.coyote.http11.AbstractHttp11Processor.prepareResponse(AbstractHttp11Processor.java:926) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:739) at org.apache.coyote.Response.action(Response.java:170) at org.apache.coyote.Response.sendHeaders(Response.java:350) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:308) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:275) at org.apache.catalina.connector.Response.finishResponse(Response.java:501) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:243) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:188) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:166) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:288) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space at java.lang.reflect.Array.newArray(Native Method) at java.lang.reflect.Array.newInstance(Unknown Source) at java.util.AbstractCollection.toArray(Unknown Source) at org.apache.catalina.session.ManagerBase.findSessions(ManagerBase.java:705) at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:527) at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:518) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1214) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1393) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1403) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1403) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1382) at java.lang.Thread.run(Unknown Source) === Thanks, Sunil. -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Monday, May 30, 2011 7:25 PM To: Tomcat Users List Subject: Re: Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space You are kidding us, right ? But just in
Re: Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space
You are kidding us, right ? But just in case you are not, what is the part which you do not understand in "java.lang.OutOfMemoryError: Java heap space". ? http://lmgtfy.com/?q=java.lang.OutOfMemoryError:%20Java%20heap%20space sunil.sheva...@wipro.com wrote: Hi, I am getting an error as follows "Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space". I am running my jsp application on a windows machine with Tomcat 7.0.8 Any pointers? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space
Hi, I am getting an error as follows "Exception in thread ""http-bio-8080"-exec-9" java.lang.OutOfMemoryError: Java heap space". I am running my jsp application on a windows machine with Tomcat 7.0.8 Any pointers? Thanks in advance, Sunil. Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: java.lang.OutOfMemoryError: Java heap space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, Another correction to the code: On 10/8/2009 4:22 PM, Christopher Schultz wrote: > public static class SoftReferenceSessionWrapper > implements HttpSession > { Remember to implement getValue and putValue to call getAttribute and setAttribute. ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrOTdUACgkQ9CaO5/Lv0PCIDwCgt+kyda0PTi+LoqQvQhGb2yHC HKMAoL/uMesdmV1Ubrco5Hyx8F9x0IKF =53/Y -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, A correction to my code: On 10/8/2009 4:22 PM, Christopher Schultz wrote: > public class SoftReferenceFilter > implements Filter > { This class needs two additional methods: public void init(FilterChain chain) { } public void destroy() { } > // Here's where the magic happens > public Object getAttribute(String key) > { > Object o = super.getAttribute(key); // Get the stored object This should be: Object o = _base.getAttribute(key); > public Object setAttribute(String key, Object value) > { > // Wrap the value in a SoftReference if appropriate > if(key.startsWith("foto") && null != value) > value = new SoftReference(value); > > super.setAttribute(key, value); This should be: _base.setAttribute(key, value); Not too bad for code typed directly into an email message. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrOTIwACgkQ9CaO5/Lv0PAgoACgoAZOVSzVhSOwD893/qC/YtZ/ 57MAnApZ064qpwt15QzMuEswaU3KFxMY =nno+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joe, On 10/8/2009 12:12 PM, Joe Hansen wrote: > I will now think twice before I store anything in the session and > will make sure I remove no longer used objects from the HTTP > Session. It's always a good goal to limit your session size, but it's not always foolproof because users don't always do the things you expect. For instance, if you have a process your user goes through, and you store a big object (or set of objects) in the session during that process, and the user does not finish the process, you still need to clean-up after that. >> You can couple session-object expiration with lazy >> instantiation/fetch for a solution that can keep your sessions >> lightweight yet allow insanely long session timeouts. > > Chris, can you please elaborate what you mean by coupling > session-object expiration with lazy fetch? So, lazy instantiation (or initialization) is a design pattern where you only create objects when you are actually going to use them, as opposed to creating everything you /might/ need just in case it's necessary. "Lazy fetch" is just a term I used to describe grabbing your Foto objects only when actually needed (rather than speculatively loading them). This concept is illustrated easily in a few lines of code: // This would be speculative instantiation Vector v = new Vector(1000); // Just in case ... if(something) { // use the Vector object } // This would be lazy instantiation Vector v = null; if(something) { if(null == v) v = new Vector(); // use the Vector object } It sounds simple, and it is. The same concept can be applied when fetching large objects from the database and possibly storing them in the session. You can use the session as a poor-mans cache something like this: // You probably already do this kind of thing in your code: Foto foto = (Foto)session.getAttribute("myFotoObject"); // But now, you can plan for the case where the object either // was never there, or has disappeared because it has been // removed to save memory: if(null == foto) { foto = fotoManager.fetch(whatever); session.setAttribute("myFotoObject", foto); } Using this technique, you can get the benefit of session "caching" that is tolerant of disappearing objects due to the "lazy-fetching" technique I describe. See below for how to hook it up to cache-flushing schemes. >> Is it an absolute requirement that these objects be stored in the >> session at all? If so, then maybe the caching strategy can be tweaked a >> bit to allow both requirements to peacefully coexist. > > The code is 3 years old and it does not use a caching strategy at all. > However, I am planning to use ehcache when we developer our future > websites. 3 years is not a ripe old age for code. Our best code is the stuff that has lasted for 3 or more years without having to be rewritten. ;) If ehcache is your thing, go for it. There are other possibilities, too. >> There are even techniques that will allow your session objects to expire >> when memory gets tight (which is super cool IMO). One way to retrofit your web application is to use SoftReferences to store objects within your session attributes. You can either re-write all your code to deal with SoftReferences (see examples below), or you can get tricky and write a simple wrapper around your request/session objects to do the magic for you. I'll show you how such tricks could be implemented. Let's say that you always use the prefix "foto" for all Foto objects in the session. You could use this technique with all session objects, but it really only makes sense with the big ones. Here's the plan: 1. Alter your code that deals with Foto objects stored in the session to tolerate the objects apparently disappearing from the session without notice. That is, always check for null and re-fetch from the database or wherever you get your Foto data. 2. SoftReference objects will be used to store your actual Foto objects in the session. 3. Write a Filter which wraps the HttpServletRequest object to return a wrapped HttpSession object which will handle the SoftReferences mentioned in #2 (got all that?) It's easier than it sounds. Let's assume #1 is already done. Here's how to write the filter (which implements both #2 and #2 above): public class SoftReferenceFilter implements Filter { public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws ServletException, IOException { if(request instanceof HttpServletRequest) request = new SoftReferenceRequestWrapper((HttpServletRequest)request); chain.doFilter(request, response); } public static class SoftReferenceRequestWrapper extends HttpServletRequestWrapper { public SoftReferenceRequestWrapper(HttpServletRequest request) { super(request); } // Override public HttpSession getSession() { HttpSession session = super.getSession();
RE: java.lang.OutOfMemoryError: Java heap space
> From: Joe Hansen [mailto:joe.hansen...@gmail.com] > Subject: Re: java.lang.OutOfMemoryError: Java heap space > > Is the use of SoftReference popular? You'll find it as the underlying mechanism of many cache managers, since it automatically adapts reasonably well to whatever the execution environment happens to be. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
> Yup, it looks like you have more of your application objects taking up > memory than the tickets you were worried about. So True! We are right now designing and developing our next generation of websites and this experience will surely come handy. I will now think twice before I store anything in the session and will make sure I remove no longer used objects from the HTTP Session. > You can couple session-object expiration with lazy instantiation/fetch for a > solution > that can keep your sessions lightweight yet allow insanely long session > timeouts. Chris, can you please elaborate what you mean by coupling session-object expiration with lazy fetch? > Is it an absolute requirement that these objects be stored in the > session at all? If so, then maybe the caching strategy can be tweaked a > bit to allow both requirements to peacefully coexist. The code is 3 years old and it does not use a caching strategy at all. However, I am planning to use ehcache when we developer our future websites. > There are even techniques that will allow your session objects to expire > when memory gets tight (which is super cool IMO). I did not know that something like that existed. Thanks for letting me know, Chris! Thanks for telling me about SoftReference, Charles. Looks like SoftReference existed since JDK 1.2 and I never knew about it! Is the use of SoftReference popular? Thanks guys! Joe - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: java.lang.OutOfMemoryError: Java heap space
> From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Subject: Re: java.lang.OutOfMemoryError: Java heap space > > There are even techniques that will allow your session objects to > expire when memory gets tight (which is super cool IMO). To put a name on the above technique, read up on using SoftReference objects to track entries in your cache(s). These will allow GC to discard objects not in immediate use when the heap is full. http://www.ibm.com/developerworks/java/library/j-jtp01246.html http://java.sun.com/javase/6/docs/api/java/lang/ref/SoftReference.html - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: java.lang.OutOfMemoryError: Java heap space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joe, On 10/7/2009 8:40 PM, Joe Hansen wrote: > WoW! "jmap" was a great command to learn Chris! Thank you so much! Yeah, it's great as a poor-man's memory profiler. On the other hand, I haven't seen a big-time memory profiler offer such a simple view of everything since Borland's OptimizeIt. > As you've pointed out, most of the data is taken by character arrays, > Strings, integer arrays, byte arrays, Foto and FotoSet beans! There > were just 67 Ticket entries (related to CAS) and the size of the > Ticket objects wasn't huge either. Yup, it looks like you have more of your application objects taking up memory than the tickets you were worried about. > I had changed the session-timeout in server.xml from 30 minutes to 240 > minutes. That means an HTTP Session would now last 8 times longer. > Since the Garbage Collector runs only when the session expires, the > application's memory needs are probably increased 8 fold (Am I > right?). Er, the GC runs all the time, but the objects in the session are not cleaned-out until the session expires. You might want to consider "cleaning" your sessions a bit more regularly. You can couple session-object expiration with lazy instantiation/fetch for a solution that can keep your sessions lightweight yet allow insanely long session timeouts. It will require you to do some additional work, but improved stability is probably worth it. > Should I just decrease the session-timeout to 2 hours and see if 512MB > is sufficient? Any other thoughts/ideas guys? I think your session timeout should be dictated by your business requirements, not technical limitations (since this is a relatively low technical hurdle). Is it an absolute requirement that these objects be stored in the session at all? If so, then maybe the caching strategy can be tweaked a bit to allow both requirements to peacefully coexist. There are even techniques that will allow your session objects to expire when memory gets tight (which is super cool IMO). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrNQGUACgkQ9CaO5/Lv0PD1XwCePdhJbbdSakmn2x9uopHrN7Na 6R8AoMDfMpMioh/J2D+UfpJTUxFiHJ3G =avdi -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
WoW! "jmap" was a great command to learn Chris! Thank you so much! I generated a histogram of the Objects in the java heap (while things were stable) and here's what I got: Object Histogram: SizeCount Class description --- 149989048 3496198 char[] 847084323529518 java.lang.String 58308016560654 tw.beans.TWFotoBean 35620304179282 byte[] 2328687230767 int[] 18220184142094 * ConstMethodKlass 14932056622169 java.sql.Date 10235120142094 * MethodKlass 1016478490757 tw.beans.TWFotoSetBean 8878248 134561 java.lang.Object[] 7949144 16190 * ConstantPoolKlass 6147768 16190 * InstanceKlassKlass 4590176 13997 * ConstantPoolCacheKlass ... ... As you've pointed out, most of the data is taken by character arrays, Strings, integer arrays, byte arrays, Foto and FotoSet beans! There were just 67 Ticket entries (related to CAS) and the size of the Ticket objects wasn't huge either. Seeing the histogram results was an eye opener. I am wondering if Rainer was right all along. Is it just the application data (FotoBean and FotoSetBean) that's inundating the java heap? I had changed the session-timeout in server.xml from 30 minutes to 240 minutes. That means an HTTP Session would now last 8 times longer. Since the Garbage Collector runs only when the session expires, the application's memory needs are probably increased 8 fold (Am I right?). And I am storing the FotoBeans and FotoSetBeans in each user's HTTP Session. As I mentioned before, previously the java heap was set to the default, 64MB, and it used to be sufficient. From the time I changed the session-timeout, even 512MB (= 64 X 8) is insufficient. Should I just decrease the session-timeout to 2 hours and see if 512MB is sufficient? Any other thoughts/ideas guys? Thanks again Chris, Joe Here are all the Ticket entries in the heap: 96 4 org.jasig.cas.util.DefaultUniqueTicketIdGenerator 72 3 org.jasig.cas.util.DefaultUniqueTicketIdGenerator 72 3 org.jasig.cas.util.DefaultUniqueTicketIdGenerator 72 3 org.jasig.cas.util.DefaultUniqueTicketIdGenerator 72 3 org.jasig.cas.util.DefaultUniqueTicketIdGenerator 72 3 org.jasig.cas.util.DefaultUniqueTicketIdGenerator 24 1 org.jasig.cas.web.flow.SendTicketGrantingTicketAction 24 1 org.jasig.cas.web.flow.SendTicketGrantingTicketAction 24 1 org.jasig.cas.web.flow.SendTicketGrantingTicketAction 24 1 org.jasig.cas.web.flow.SendTicketGrantingTicketAction 24 1 org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator 24 1 org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator 24 1 org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator 24 1 org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator 24 1 org.jasig.cas.web.flow.SendTicketGrantingTicketAction 24 1 org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator 24 1 org.jasig.cas.web.flow.SendTicketGrantingTicketAction 24 1 org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator 16 1 org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator 16 1 org.jasig.cas.ticket.registry.DefaultTicketRegistry 16 1 org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner 16 1 org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator 16 1 org.jasig.cas.ticket.registry.DefaultTicketRegistry 16 1 org.acegisecurity.providers.cas.proxy.RejectProxyTickets 16 1 org.jasig.cas.web.flow.GenerateServiceTicketAction 16 1 org.jasig.cas.ticket.registry.DefaultTicketRegistry 16 1 org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner 16 1 org.acegisecurity.providers.cas.proxy.RejectProxyTickets 16 1 org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner 16 1 org.jasig.cas.web.flow.GenerateServiceTicketAction 16 1 org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator 16 1 org.acegisecurity.providers.cas.cache.EhCacheBasedTicketCache 16 1 org.acegisecurity.providers.cas.cache.EhCacheBasedTicketCache 16 1 org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner 16 1 org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator 16 1 org.jasig.cas.web.flow.GenerateServiceTicketAction 16 1 org.acegisecurity.providers.cas.proxy.RejectProxyTickets 16 1 org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner 16 1 org.jasig.cas.web.flow.GenerateServiceTicketAction 16 1 org.acegisecurity.providers.cas.proxy.RejectProxyTickets 16 1 org.jasig.cas.ticket.registry.DefaultTicketRegistry 16
Re: java.lang.OutOfMemoryError: Java heap space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joe, On 10/6/2009 3:44 PM, Joe Hansen wrote: > I did not check the CPU usage when the issue happened. I just checked > the web application logs and found that first there was an > OutOfMemoryError, followed by MySQL errors when > TWFotoSetListDAO.getAllFotosets tried to get a database connection. It's possible that, if you really are grabbing photos (like, actual images) from a database and fitting them into memory, you might just be busting your heap yourself (rather than CAS, etc.). Are there large objects that you store in memory and/or sessions? > I am *guessing* : > I changed the session timeout to 4 hours for the Jasig CAS > Single-Signon web application. This stores a ton of tickets in memory. > During a 4 hour period when there are more http requests, the 512MB of > Java Heap space is not sufficient to store all these tickets. During > that time OutOfMemoryError happens and all hell breaks loose > subsequently. Wow, those "tickets" seem very heavyweight. Try this when the server is "quiet" and then, later, when it's getting stormed: $ jmap -histo [jvm pid] > heap.log Then, go into the heap log and see what's taking up so much of your memory. For instance, if I run this against my own JVM running in development, here are the top 10 objects in memory: SizeCount Class description - --- 1078520098394 char[] 5548488 40283 * ConstMethodKlass 2911352 7238byte[] 2465208 102717 java.lang.String 2259496 40283 * MethodKlass 2212736 54518 * SymbolKlass 2171032 3767* ConstantPoolKlass 1912032 7936int[] 1761184 38585 java.lang.Object[] 1500112 3767* InstanceKlassKlass These are the objects taking up most memory. If you look at the "count" column, you can see how many of the objects are in memory. I suggest looking at both the "size" and the "count" columns, because your understanding of the "size" of an object and the JVM's notion may differ: most heap space is taken up by byte and char arrays and things like that, but then encapsulated into "larger" objects that, themselves, take up very little space. You may find that you have thousands of CAS tickets in memory and that they are very heavy. You may also find that the tickets themselves aren't the problem, but that a 4-hour session timeout leads to sessions with huge amounts of cached information in them. It's possible that your webapp is simply being careless with memory if you want a 4-hour session timeout. You might want to invest in a real memory profiler to see what is happening. These products can give you a better picture of "large" objects that you can look at to make decisions as to where you will apply your efforts. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrM2wYACgkQ9CaO5/Lv0PCjTwCfS/bwZmXLT+BUgYxJVyqIv6tT AuYAn1Jupcl6eB240BHH3pvtKJNGTglq =XDLj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
2009/10/6 Joe Hansen : > Andre, you are right. Memory is cheap. However, the machine has 8GB of > memory already. I am thinking that it should be sufficient. You might wish to dedicate more of that 8 Gbyte to Java, though. It depends on the performance characteristics of your application - looks to me like MySQL is grabbing a load of your RAM for its cache. top (or similar) would confirm that overall usage, and I assume MySQL has some performance monitoring available that'll tell you the effectiveness of its caches. If you're *just* tipping into using swap, it's almost certainly the case that MySQL is grabbing memory that would otherwise be called "free". This may well be the best use for the RAM, but there's a possibility that more RAM dedicated to Java would improve your performance overall. I'm not willing to make that guess without data on the effectiveness of MySQL's cache, though :-). - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
I changed the session-timeout for the CAS webapp to 1 hour and restarted Apache/Tomcat to see if that changes anything. If the issue manifests again, I will surely check the memory and cpu usage using the top command. Rainer, I somewhat doubt that getAllFotosets() is causing the problem. Because I have an administrator account on the website and I can view more photosets than any other user. And when I login and view the photosets nothing alarming seems to be happening. I am not ruling out the possibility altogether though. Andre, you are right. Memory is cheap. However, the machine has 8GB of memory already. I am thinking that it should be sufficient. Here's the output of the free command: [r...@csiweb bin]# free -m totalused free Mem: 8113 7557 555 -/+ buffers/cache:861 7252 Swap:9012 33 8979 Its amazing how you and Andre are able to identify issues with others' Tomcat installations just by looking at their logs and settings. It speaks volumes of your experience with Tomcat. Thank you, Joe On Tue, Oct 6, 2009 at 2:00 PM, Rainer Jung wrote: > On 06.10.2009 21:44, Joe Hansen wrote: >> Thanks for your feedback, Rainer. >> >> The DAO objects/methods that you've mentioned are used for almost >> every page on our website. So I am not surprised to find them in the >> thread dumps. However, as you pointed out, the parameters to these >> methods change depending on the page they are on and if the user is >> logged in or not. >> >> I did not check the CPU usage when the issue happened. I just checked >> the web application logs and found that first there was an >> OutOfMemoryError, followed by MySQL errors when >> TWFotoSetListDAO.getAllFotosets tried to get a database connection. >> >> I am *guessing* : >> I changed the session timeout to 4 hours for the Jasig CAS >> Single-Signon web application. This stores a ton of tickets in memory. >> During a 4 hour period when there are more http requests, the 512MB of >> Java Heap space is not sufficient to store all these tickets. During >> that time OutOfMemoryError happens and all hell breaks loose >> subsequently. >> >> I probably should decrease the session-timeout to 1 hour or so and see >> if that changes things. Would you agree, Rainer? > > Could be. But it could also be, that getAllFotosets() is able to slurp > in huge amounts of data to memory and that this is the reason for the OOME. > > Once you run into an OOME on heap, there's no good relief than > restarting the whole JVM. After an OOME you can't prove the correctness > of your app any more. Random objects might be missing, resulting in > largely undefined behaviour in your webapp or even Tomcat. > > I'm a bit insisting on the getAllFotosets(), because I once was involved > in troubleshooting for a social website and part of their problem was > special users having extremely large social data. So once they hit the > server it went OOME. > >> Thanks, >> Joe > > Regards, > > Rainer > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Rainer Jung wrote: On 06.10.2009 21:44, Joe Hansen wrote: ... It seems your application is CPU heavy. Either the data objects handled are to heavy weight (maybe some user having huge Fotoset or Email list) or the request rate is simply to large. Is the CPU saturated during the problems? I would activate a good access log and try to find out from that and your webapp logs what maybe special about these web requests or users. ... The original post was so long ago in relative terms, that I don't remember the details of your system. But, just in case, you may want to also have a look at the *total* memory usage on your system during those events. If the load has increased (even slightly) recently, it might have reached the point where there is sometimes no longer sufficient physical memory available to process all simultaneous tasks, and the system is starting to swap tasks to virtual memory (on disk). That would cause a dramatic slowdown in request processing, which may have something to do with your problems. If you are under Unix/Linux, the first lines displayed by "top" would already provide some information in that respect. Or else, just plug in an additional memory bar in your server, and see if it changes anything. They are rather cheap right now. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
On 06.10.2009 21:44, Joe Hansen wrote: > Thanks for your feedback, Rainer. > > The DAO objects/methods that you've mentioned are used for almost > every page on our website. So I am not surprised to find them in the > thread dumps. However, as you pointed out, the parameters to these > methods change depending on the page they are on and if the user is > logged in or not. > > I did not check the CPU usage when the issue happened. I just checked > the web application logs and found that first there was an > OutOfMemoryError, followed by MySQL errors when > TWFotoSetListDAO.getAllFotosets tried to get a database connection. > > I am *guessing* : > I changed the session timeout to 4 hours for the Jasig CAS > Single-Signon web application. This stores a ton of tickets in memory. > During a 4 hour period when there are more http requests, the 512MB of > Java Heap space is not sufficient to store all these tickets. During > that time OutOfMemoryError happens and all hell breaks loose > subsequently. > > I probably should decrease the session-timeout to 1 hour or so and see > if that changes things. Would you agree, Rainer? Could be. But it could also be, that getAllFotosets() is able to slurp in huge amounts of data to memory and that this is the reason for the OOME. Once you run into an OOME on heap, there's no good relief than restarting the whole JVM. After an OOME you can't prove the correctness of your app any more. Random objects might be missing, resulting in largely undefined behaviour in your webapp or even Tomcat. I'm a bit insisting on the getAllFotosets(), because I once was involved in troubleshooting for a social website and part of their problem was special users having extremely large social data. So once they hit the server it went OOME. > Thanks, > Joe Regards, Rainer > 2009 Oct 06 / 10:48:43 ERROR - > [org.apache.catalina.core.ContainerBase] : Exception invoking periodic > operation: > java.lang.OutOfMemoryError: Java heap space > > 2009 Oct 06 / 10:48:43 FATAL - [tw.conn.TWConnectionManager] : > com.mysql.jdbc.CommunicationsException: Communications link failure > due to underlying exception: > > ** BEGIN NESTED EXCEPTION ** > > java.io.EOFException > > STACKTRACE: > > java.io.EOFException > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771) > at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3649) > at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1176) > at com.mysql.jdbc.Connection.createNewIO(Connection.java:2558) > at com.mysql.jdbc.Connection.(Connection.java:1485) > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) > at > org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37) > at > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) > at > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771) > at > org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) > at > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) > at > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) > at tw.conn.TWConnectionManager.getConnection(Unknown Source) > at tw.beans.TWFotoSetListDAO.getAllFotosets(Unknown Source) > at tw.beans.TWViewBean.getAllFotosets(Unknown Source) > at tw.servlet.TWQueryServlet.doPost(Unknown Source) > at tw.servlet.TWQueryServlet.doGet(Unknown Source) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) > at > org.apache.jsp.groupby_005fitems_jsp._jspService(groupby_005fit
Re: java.lang.OutOfMemoryError: Java heap space
Thanks for your feedback, Rainer. The DAO objects/methods that you've mentioned are used for almost every page on our website. So I am not surprised to find them in the thread dumps. However, as you pointed out, the parameters to these methods change depending on the page they are on and if the user is logged in or not. I did not check the CPU usage when the issue happened. I just checked the web application logs and found that first there was an OutOfMemoryError, followed by MySQL errors when TWFotoSetListDAO.getAllFotosets tried to get a database connection. I am *guessing* : I changed the session timeout to 4 hours for the Jasig CAS Single-Signon web application. This stores a ton of tickets in memory. During a 4 hour period when there are more http requests, the 512MB of Java Heap space is not sufficient to store all these tickets. During that time OutOfMemoryError happens and all hell breaks loose subsequently. I probably should decrease the session-timeout to 1 hour or so and see if that changes things. Would you agree, Rainer? Thanks, Joe 2009 Oct 06 / 10:48:43 ERROR - [org.apache.catalina.core.ContainerBase] : Exception invoking periodic operation: java.lang.OutOfMemoryError: Java heap space 2009 Oct 06 / 10:48:43 FATAL - [tw.conn.TWConnectionManager] : com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception: ** BEGIN NESTED EXCEPTION ** java.io.EOFException STACKTRACE: java.io.EOFException at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771) at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3649) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1176) at com.mysql.jdbc.Connection.createNewIO(Connection.java:2558) at com.mysql.jdbc.Connection.(Connection.java:1485) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) at org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37) at org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771) at org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) at tw.conn.TWConnectionManager.getConnection(Unknown Source) at tw.beans.TWFotoSetListDAO.getAllFotosets(Unknown Source) at tw.beans.TWViewBean.getAllFotosets(Unknown Source) at tw.servlet.TWQueryServlet.doPost(Unknown Source) at tw.servlet.TWQueryServlet.doGet(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) at org.apache.jsp.groupby_005fitems_jsp._jspService(groupby_005fitems_jsp.java:152) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
Re: java.lang.OutOfMemoryError: Java heap space
On 06.10.2009 19:44, Joe Hansen wrote: I will only comment the threads for the AJP pool. Everything else seems not relevant: Dump 1: 20 threads connected to Apache, waiting for the next request 3 threads idle in the pool 1 thread waiting for the next connection 0 threads working on requests Dump 2: 12 threads connected to Apache, waiting for the next request 5 threads idle in the pool 1 thread waiting for the next connection 14 threads working on requests Dump 3: 11 threads connected to Apache, waiting for the next request 5 threads idle in the pool 1 thread waiting for the next connection 15 threads working on requests Dump 4: 11 threads connected to Apache, waiting for the next request 5 threads idle in the pool 1 thread waiting for the next connection 15 threads working on requests Those busy threads in dump 2, 3 and 4 are the same except for one, which started only in dump 3. Most of them are busy working on data. They are *not* waiting for some other system, like database or similar. Out of the 14+15+15 thread stacks, that are interesting, 42 work on some DAOs. Those are the DAO methods they work on: 18 tw.beans.TWFotoSetDAO.getFotosetFotos 11 tw.beans.TWEmailUpdateListDAO.getEmailUpdateList 3 tw.beans.TWFotoSetListDAO.getAllFotosets 3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery2 3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery 3 tw.beans.TWEmailUpdateDAO.getEmailUpdate 1 tw.beans.TWEmailUpdateListDAO.getEmailUpdateListQuery It seems your application is CPU heavy. Either the data objects handled are to heavy weight (maybe some user having huge Fotoset or Email list) or the request rate is simply to large. Is the CPU saturated during the problems? I would activate a good access log and try to find out from that and your webapp logs what maybe special about these web requests or users. Regards, Rainer > I will learn from your previous analysis of the thread dumps and I try > to understand what's happening. > > Thanks, > Joe > > On Tue, Oct 6, 2009 at 10:23 AM, Joe Hansen wrote: >> Rainer, >> >> Thanks for looking at those long thread dumps for me!! >> >> I am sorry. I did NOT take these dumps at the right time (i.e. when >> Tomcat was inundated with requests and couldn't cope with the load). >> After I increased the heap size to 512MB (from 64MB default), I am not >> getting the OutOfMemoryError(s) anymore. After I set KeepAlive On >> (Thanks Andre!), the number httpd processes isn't increasing either. >> The number of httpd processes increased from 8 to 21 and stayed there >> for more than 16 hours now. If the number of httpd processes gets out >> of control again, I will definitely take thread dumps once again. >> >> However, I doubt the issue is fixed because I should have seen this >> issue long time ago (since I haven't changed any code nor the traffic >> to our websites increased in a long while). >> >> Should I just wait and see or are there any tests that I can do? >> >> Your contribution to this forum is amazing, Rainer. I am grateful to >> you and Andre for your efforts. Thank you! >> >> Regards, >> Joe >> >> >> >> On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung wrote: >>> On 05.10.2009 18:58, Joe Hansen wrote: Thank you so much for your tips, Rainer! The websites went down yet again. Increasing the java heap size took care of the OutOfMemoryError, but the number of httpd processes keep increasing until the websites crash. I haven't added any new code in the past few months, hence I am surprised why the requests are getting stuck. Here's a link to the tomcat thread dumps: http://pastebin.com/m17eea139 Please let me know if you cannot view it and I will email the relevant portion of the catalina.out file to you. Is there an easy way to find out what code is causing the requests to get stuck? >>> >>> The dump file contains three thread dumps. >>> >>> The things all dumps have in common: >>> >>> - 60 threads for the quartz scheduler, all idle >>> - 13 threads in the AJP connection pool, connected to Apache, but idle >>> waiting for the next request to be send (the same threads in all three >>> dumps) >>> - 6 store plus 6 expiry threads of the EHCache, seems idle >>> - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new >>> connection to come in >>> - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting >>> for work >>> - a couple of other normal threads not directly related to request handling >>> >>> So the time you took the three dumps, this Tomcat was completely idle >>> and did not have a single request to handle. >>> >>> If you are completely sure, you took the dumps while there was a storm >>> of requests and your system couldn't cope the load, something has >>> prevented the requests to ever reach Tomcat. >>> >>> I don't have your Tomcat version at hand at the moment, but for some >>> time very special OutOfMemory errors (could not create native thread) >>> l
Re: java.lang.OutOfMemoryError: Java heap space
There were 29 httpd processes running when the websites crashed. One thing that was clear from the thread dumps was there were lots of entries for org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner: Starting cleaning of expired tickets from ticket registry at ... Does that indicate that there is something wrong with my CAS configuration? Here's the thread count from the first thread dump: DefaultQuartzScheduler_QuartzSchedulerThread 4 sleeping + 2 waiting on condition = 6 Total DefaultQuartzScheduler_Worker 60 waiting = 60 Total Store ticketCache Expiry Thread (ehcache) 6 waiting on condition = 6 Total Store ticketCache Spool Thread (ehcache) 6 waiting on condition = 6 Total TP-ProcessorXY daemon runnable 20 runnable (mod_jk?) + 3 waiting = 23 Total Java2D Disposer daemon 1 waiting = 1 Total http-8443-Monitor 1 waiting = 1 Total http-8443-Processor 3 waiting + 1 runnable = 4 Total ContainerBackgroundProcessor 1 waiting on condition = 1 Total Low Memory Detector 1 runnable = 1 Total CompilerThread 2 waiting on condition = 2 Total AdapterThread 1 waiting on condition = 1 Total Signal Dispatcher 1 runnable = 1 Total Finalizer 1 waiting = 1 Total Reference Handler 1 waiting = 1 Total org.apache.catalina.startup.Bootstrap.main 1 runnable = 1 Total VM Thread 1 runnable = 1 Total GC task thread 1 runnable = 1 Total VM Periodic Task Thread 1 waiting on condition = 1 Total Thanks, Joe On Tue, Oct 6, 2009 at 11:44 AM, Joe Hansen wrote: > Rainer, > > I spoke to soon! As I suspected, the problem isn't fixed yet and the > websites crashed again. This time I took three thread dumps at the > time tomcat was down. Here they are: http://pastebin.com/m2a7e1198 > > I will learn from your previous analysis of the thread dumps and I try > to understand what's happening. > > Thanks, > Joe > > On Tue, Oct 6, 2009 at 10:23 AM, Joe Hansen wrote: >> Rainer, >> >> Thanks for looking at those long thread dumps for me!! >> >> I am sorry. I did NOT take these dumps at the right time (i.e. when >> Tomcat was inundated with requests and couldn't cope with the load). >> After I increased the heap size to 512MB (from 64MB default), I am not >> getting the OutOfMemoryError(s) anymore. After I set KeepAlive On >> (Thanks Andre!), the number httpd processes isn't increasing either. >> The number of httpd processes increased from 8 to 21 and stayed there >> for more than 16 hours now. If the number of httpd processes gets out >> of control again, I will definitely take thread dumps once again. >> >> However, I doubt the issue is fixed because I should have seen this >> issue long time ago (since I haven't changed any code nor the traffic >> to our websites increased in a long while). >> >> Should I just wait and see or are there any tests that I can do? >> >> Your contribution to this forum is amazing, Rainer. I am grateful to >> you and Andre for your efforts. Thank you! >> >> Regards, >> Joe >> >> >> >> On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung wrote: >>> On 05.10.2009 18:58, Joe Hansen wrote: Thank you so much for your tips, Rainer! The websites went down yet again. Increasing the java heap size took care of the OutOfMemoryError, but the number of httpd processes keep increasing until the websites crash. I haven't added any new code in the past few months, hence I am surprised why the requests are getting stuck. Here's a link to the tomcat thread dumps: http://pastebin.com/m17eea139 Please let me know if you cannot view it and I will email the relevant portion of the catalina.out file to you. Is there an easy way to find out what code is causing the requests to get stuck? >>> >>> The dump file contains three thread dumps. >>> >>> The things all dumps have in common: >>> >>> - 60 threads for the quartz scheduler, all idle >>> - 13 threads in the AJP connection pool, connected to Apache, but idle >>> waiting for the next request to be send (the same threads in all three >>> dumps) >>> - 6 store plus 6 expiry threads of the EHCache, seems idle >>> - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new >>> connection to come in >>> - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting >>> for work >>> - a couple of other normal threads not directly related to request handling >>> >>> So the time you took the three dumps, this Tomcat was completely idle >>> and did not have a single request to handle. >>> >>> If you are completely sure, you took the dumps while there was a storm >>> of requests and your system couldn't cope the load, something has >>> prevented the requests to ever reach Tomcat. >>> >>> I don't have your Tomcat version at hand at the moment, but for some >>> time very special OutOfMemory errors (could not create native thread) >>> lead to a situation, where Tomcat simply wouldn't accept any new >>> connections. Although you report OutOfMemory errors, I'm not directly >>> suggesting that that is
Re: java.lang.OutOfMemoryError: Java heap space
Rainer, I spoke to soon! As I suspected, the problem isn't fixed yet and the websites crashed again. This time I took three thread dumps at the time tomcat was down. Here they are: http://pastebin.com/m2a7e1198 I will learn from your previous analysis of the thread dumps and I try to understand what's happening. Thanks, Joe On Tue, Oct 6, 2009 at 10:23 AM, Joe Hansen wrote: > Rainer, > > Thanks for looking at those long thread dumps for me!! > > I am sorry. I did NOT take these dumps at the right time (i.e. when > Tomcat was inundated with requests and couldn't cope with the load). > After I increased the heap size to 512MB (from 64MB default), I am not > getting the OutOfMemoryError(s) anymore. After I set KeepAlive On > (Thanks Andre!), the number httpd processes isn't increasing either. > The number of httpd processes increased from 8 to 21 and stayed there > for more than 16 hours now. If the number of httpd processes gets out > of control again, I will definitely take thread dumps once again. > > However, I doubt the issue is fixed because I should have seen this > issue long time ago (since I haven't changed any code nor the traffic > to our websites increased in a long while). > > Should I just wait and see or are there any tests that I can do? > > Your contribution to this forum is amazing, Rainer. I am grateful to > you and Andre for your efforts. Thank you! > > Regards, > Joe > > > > On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung wrote: >> On 05.10.2009 18:58, Joe Hansen wrote: >>> Thank you so much for your tips, Rainer! >>> >>> The websites went down yet again. Increasing the java heap size took >>> care of the OutOfMemoryError, but the number of httpd processes keep >>> increasing until the websites crash. I haven't added any new code in >>> the past few months, hence I am surprised why the requests are getting >>> stuck. Here's a link to the tomcat thread dumps: >>> http://pastebin.com/m17eea139 >>> >>> Please let me know if you cannot view it and I will email the relevant >>> portion of the catalina.out file to you. Is there an easy way to find >>> out what code is causing the requests to get stuck? >> >> The dump file contains three thread dumps. >> >> The things all dumps have in common: >> >> - 60 threads for the quartz scheduler, all idle >> - 13 threads in the AJP connection pool, connected to Apache, but idle >> waiting for the next request to be send (the same threads in all three >> dumps) >> - 6 store plus 6 expiry threads of the EHCache, seems idle >> - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new >> connection to come in >> - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting >> for work >> - a couple of other normal threads not directly related to request handling >> >> So the time you took the three dumps, this Tomcat was completely idle >> and did not have a single request to handle. >> >> If you are completely sure, you took the dumps while there was a storm >> of requests and your system couldn't cope the load, something has >> prevented the requests to ever reach Tomcat. >> >> I don't have your Tomcat version at hand at the moment, but for some >> time very special OutOfMemory errors (could not create native thread) >> lead to a situation, where Tomcat simply wouldn't accept any new >> connections. Although you report OutOfMemory errors, I'm not directly >> suggesting that that is your problem here. There might still be a >> relation though. >> >> Are you sure, that you took the dumps for the right Tomcat at the right >> time? >> >> Regards, >> >> Rainer >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Rainer, Thanks for looking at those long thread dumps for me!! I am sorry. I did NOT take these dumps at the right time (i.e. when Tomcat was inundated with requests and couldn't cope with the load). After I increased the heap size to 512MB (from 64MB default), I am not getting the OutOfMemoryError(s) anymore. After I set KeepAlive On (Thanks Andre!), the number httpd processes isn't increasing either. The number of httpd processes increased from 8 to 21 and stayed there for more than 16 hours now. If the number of httpd processes gets out of control again, I will definitely take thread dumps once again. However, I doubt the issue is fixed because I should have seen this issue long time ago (since I haven't changed any code nor the traffic to our websites increased in a long while). Should I just wait and see or are there any tests that I can do? Your contribution to this forum is amazing, Rainer. I am grateful to you and Andre for your efforts. Thank you! Regards, Joe On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung wrote: > On 05.10.2009 18:58, Joe Hansen wrote: >> Thank you so much for your tips, Rainer! >> >> The websites went down yet again. Increasing the java heap size took >> care of the OutOfMemoryError, but the number of httpd processes keep >> increasing until the websites crash. I haven't added any new code in >> the past few months, hence I am surprised why the requests are getting >> stuck. Here's a link to the tomcat thread dumps: >> http://pastebin.com/m17eea139 >> >> Please let me know if you cannot view it and I will email the relevant >> portion of the catalina.out file to you. Is there an easy way to find >> out what code is causing the requests to get stuck? > > The dump file contains three thread dumps. > > The things all dumps have in common: > > - 60 threads for the quartz scheduler, all idle > - 13 threads in the AJP connection pool, connected to Apache, but idle > waiting for the next request to be send (the same threads in all three > dumps) > - 6 store plus 6 expiry threads of the EHCache, seems idle > - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new > connection to come in > - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting > for work > - a couple of other normal threads not directly related to request handling > > So the time you took the three dumps, this Tomcat was completely idle > and did not have a single request to handle. > > If you are completely sure, you took the dumps while there was a storm > of requests and your system couldn't cope the load, something has > prevented the requests to ever reach Tomcat. > > I don't have your Tomcat version at hand at the moment, but for some > time very special OutOfMemory errors (could not create native thread) > lead to a situation, where Tomcat simply wouldn't accept any new > connections. Although you report OutOfMemory errors, I'm not directly > suggesting that that is your problem here. There might still be a > relation though. > > Are you sure, that you took the dumps for the right Tomcat at the right > time? > > Regards, > > Rainer > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
On 05.10.2009 18:58, Joe Hansen wrote: > Thank you so much for your tips, Rainer! > > The websites went down yet again. Increasing the java heap size took > care of the OutOfMemoryError, but the number of httpd processes keep > increasing until the websites crash. I haven't added any new code in > the past few months, hence I am surprised why the requests are getting > stuck. Here's a link to the tomcat thread dumps: > http://pastebin.com/m17eea139 > > Please let me know if you cannot view it and I will email the relevant > portion of the catalina.out file to you. Is there an easy way to find > out what code is causing the requests to get stuck? The dump file contains three thread dumps. The things all dumps have in common: - 60 threads for the quartz scheduler, all idle - 13 threads in the AJP connection pool, connected to Apache, but idle waiting for the next request to be send (the same threads in all three dumps) - 6 store plus 6 expiry threads of the EHCache, seems idle - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new connection to come in - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting for work - a couple of other normal threads not directly related to request handling So the time you took the three dumps, this Tomcat was completely idle and did not have a single request to handle. If you are completely sure, you took the dumps while there was a storm of requests and your system couldn't cope the load, something has prevented the requests to ever reach Tomcat. I don't have your Tomcat version at hand at the moment, but for some time very special OutOfMemory errors (could not create native thread) lead to a situation, where Tomcat simply wouldn't accept any new connections. Although you report OutOfMemory errors, I'm not directly suggesting that that is your problem here. There might still be a relation though. Are you sure, that you took the dumps for the right Tomcat at the right time? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
On 05.10.2009 18:58, Joe Hansen wrote: > Thank you so much for your tips, Rainer! > > The websites went down yet again. Increasing the java heap size took > care of the OutOfMemoryError, but the number of httpd processes keep > increasing until the websites crash. I haven't added any new code in > the past few months, hence I am surprised why the requests are getting > stuck. Here's a link to the tomcat thread dumps: > http://pastebin.com/m17eea139 Just tried to look at it, but pastebin replies with: Down for maintenance - 6th Oct 2009 Pastebin.com is getting an unprecedented amount of traffic due to a news story in which some leaked Hotmail passwords have been pasted on this site ... Let's see, when they will be up again. They're running Apache 1.3.33 ... > Please let me know if you cannot view it and I will email the relevant > portion of the catalina.out file to you. Is there an easy way to find > out what code is causing the requests to get stuck? > > Thank you! > > Joe > > > On Sun, Oct 4, 2009 at 2:36 PM, Rainer Jung wrote: >> Hi Joe, >> >> On 04.10.2009 21:45, Joe Hansen wrote: >>> Rainer, Thank you so much for your kind reply! >>> >>> I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am >>> hoping that would fix the issue. I had configured our webserver to use >>> Jasig's Central Authentication System (CAS). Recently I increased the >>> session timeout from 30 minutes to 4 hours. I am guessing that must >>> have had an impact on the number of tickets that the CAS could store >>> in the Java's memory space. >>> >>> I did run the kill -QUIT command against the tomcat process. It did >>> generate a huge output in the catalina.out file. I am unable to >>> decipher it. I do not want to post it to the mailing list because its >>> very long. Would you be able to please tell me what should I be >>> looking for within this long thread dump? >> >> Can you put it somewhere on the web, so we can look at it, or are you >> afraid there is something private in there? You could use pastebin or >> something similar in case you do not have a public web server yourself. >> >> If you don't want to post in public, you can also mail it to me, I will >> post the result, in case I find something relevant. >> >> Regards, >> >> Rainer >> >>> On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung >>> wrote: >>>> On 03.10.2009 20:07, Joe Hansen wrote: >>>>> Hey All, >>>>> >>>>> I get this error (java.lang.OutOfMemoryError: Java heap space) after >>>>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >>>>> for a few hours. This problem started just since two days. Never had >>>>> this issue before! >>>>> >>>>> I have also noticed that as soon as I startup the server, 9 httpd >>>>> processes start. Number of httpd processes keep on increasing until I >>>>> get the OutOfMemoryError. >>>>> $ps -aef | grep httpd >>>>> root 31984 1 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31987 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31988 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31989 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31990 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31991 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31992 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31993 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>>> apache 31994 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> >>>> Sounds like requests get stuck or responses are only returned very slowly. >>>> >>>> I would take thread dumps during the time requests pile up (e.g. httpd >>>> process count increases). Thread dumps are generated by "kil -QUIT" >>>> against the Tomcat process. Result is written to catalina.out. Always >>>> take afew thread dumps shortly after each other, e.g. 3 dumps each 3 >>>> seconds apart from the previous one, so that you can find out, if a >>>> status in a dump is pure coincidence or lasts for somewhat longer. >>>> >>>>> $ps -aef | grep tomcat >>>>> root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java &
Re: java.lang.OutOfMemoryError: Java heap space
On 05.10.2009 22:04, André Warnier wrote: > André Warnier wrote: > ... > and still wants to add something : > > - a new KeepAlive connection is made from the browser to Apache (httpd). > - then a request comes in on that connection, and it happens to be one > that gets forwarded to Tomcat. So a mod_jk connection is made to > Tomcat, Tomcat allocates a thread for it. > - I would imagine that mod_jk must pass on to Tomcat the fact that this > is a KeepAlive connection, so that Tomcat would know that its thread for > that connection, should also wait for subsequent requests. > - so now the webapp/thread generates the response to the first request, > and waits on the connection for more requests. > - however, the browser does send more requests to Apache, but these are > not ones that get forwarded to Tomcat (for example, they are for items > that Apache serves locally) ... > > So now I wonder about how Apache + mod_jk + Tomcat react in such a > situation. Do the mod_jk connection +Tomcat thread keep waiting anyway, > and how long ? No in general. mod_jk only kicks in after a new request arrived over such a connection. mod_jk is agnotic about whether the rquests came over the same connection or not, even whether they belong to the same user or not. But: in case you are using a prefork MPM, each connection gets assigned to one process having one thread. This process has a connection pool of size 1 to each backend and the backend ties a dedicated thread to that connection (unless using tcnative). So in this case, it is a yes. Between two Keep Alive requests, there's no way of letting the Apache process handle other requests and thus the backend connection and backend thread are blocked waiting for the next request. That's why e.g. NTLM has a chance to work via mod_jk in combination with prefork. So here's a use case for the worker MPM, because then one client connection still blocks one Apache thread. But multiple connections are concurrently handled by the same Apache process, so mod_jk can efficiently reuse its backend connections for requests from other connections handled by the same process. The total used connection pool size might then be much smaller than with the prefork MPM. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Andre, Thanks for pointing out the high KeepAliveTimeout value in the config file. I have read the docs and have changed it to 5 seconds (which is the default). I am hoping that Rainer could find out from the thread dump where the requests are getting stuck, so that I can put this issue to bed. Thanks, Joe On Mon, Oct 5, 2009 at 1:53 PM, André Warnier wrote: > Joe Hansen wrote: >> >> Thank you for the reply, Andre. >> >> I now understand how setting KeepAlive to On would improve the >> performance of a website (The Apache manual says that a 50% increase >> in throughput could be expected). So I changed the KeepAlive to On and >> restarted the server. > > Now wait. > You should probably then lower your setting for KeepAliveTimeout (to 3 > e.g.), otherwise you may make the problem much worse. > Read conscienciously the relevant Apache doc page : > http://httpd.apache.org/docs/2.2/mod/core.html#keepalive > > The point with KeepAlive is : > - the browser makes a connection and issues a first request > - the webserver dedicates a child (or thread) to this connection, and passes > it the first request > - the child/thread responds to the first request, and then waits for more > - the browser, in the response page, finds more links. Over the same TCP > connection, it sends the next request > - the same child/thread - which was waiting on that connection - receives > the new request, and responds to it. Then it waits again for the next one. > - etc.. > - until at some point, the browser does not issue any additional requests on > the connection. Then, *after the KeepAliveTimeout has expired*, the > child/thread gives up, closesthe connection, and returns to the pool > available for other requests from other browsers > > So the point is, if the KeepAliveTimeout is long (like 15 seconds), it means > that a child/thread may be kept waiting, for nothing, up to that many > seconds, although there is nothing coming anymore. > >> >> I however wonder if this will fix the issue. The reason being, I >> haven't changed the website code at all the past few months and there >> hasn't been any increase in the website traffic too. Hence I am unable >> to understand why we are suddenly seeing an increase in the number of >> httpd processes. The only thing I changed is the session-timeout value >> from 30 minutes to 240 minutes. >> > I guess that this is the Tomcat session timeout. That should have nothing > to do with the above. I don't think that for Tomcat, a "session" is linked > to a connection. It is more of a set of data saved somewhere, linked to the > Tomcat session-id (the JSESSIONID cookie for instance). Tomcar retrieves it > whenever a request comes in with the same session-id number. But it should > not matter whether it is on the same TCP connection or not. > > What may be linked together however, is that one request to httpd results in > one child/thread busy with it at the Apache httpd level. If that request is > being forwarded to Tomcat by mod_jk, then it also holds onto one > mod_jk/Tomcat connection. This connection then holds on to one thread in > Tomcat, until the Tomcat thread (+webapp) has supplied the full response. > All the while, this whole chain is unavailable for other requests. Thus, if > there are many such requests under way, many Apache children/threads are > busy, and Apache httpd will start additional ones (up to its limit) to > service new requests that come in. > So if for some reason, your Tomcat requests now take longer to be serviced, > that should also, by retro-effect, increase the number of httpd > children/threads being started. > The bottleneck would be in Tomcat, but it would show up at the httpd level. > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: java.lang.OutOfMemoryError: Java heap space
> From: André Warnier [mailto:a...@ice-sa.com] > Subject: Re: java.lang.OutOfMemoryError: Java heap space > > The bottleneck would be in Tomcat, but it would show up at the > httpd level. The bottleneck might also be in something external to Tomcat, such as a database or some external web service used by the webapps. The result may still only be visible at the front end. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
André Warnier wrote: ... and still wants to add something : - a new KeepAlive connection is made from the browser to Apache (httpd). - then a request comes in on that connection, and it happens to be one that gets forwarded to Tomcat. So a mod_jk connection is made to Tomcat, Tomcat allocates a thread for it. - I would imagine that mod_jk must pass on to Tomcat the fact that this is a KeepAlive connection, so that Tomcat would know that its thread for that connection, should also wait for subsequent requests. - so now the webapp/thread generates the response to the first request, and waits on the connection for more requests. - however, the browser does send more requests to Apache, but these are not ones that get forwarded to Tomcat (for example, they are for items that Apache serves locally) ... So now I wonder about how Apache + mod_jk + Tomcat react in such a situation. Do the mod_jk connection +Tomcat thread keep waiting anyway, and how long ? Rainer ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Joe Hansen wrote: Thank you for the reply, Andre. I now understand how setting KeepAlive to On would improve the performance of a website (The Apache manual says that a 50% increase in throughput could be expected). So I changed the KeepAlive to On and restarted the server. Now wait. You should probably then lower your setting for KeepAliveTimeout (to 3 e.g.), otherwise you may make the problem much worse. Read conscienciously the relevant Apache doc page : http://httpd.apache.org/docs/2.2/mod/core.html#keepalive The point with KeepAlive is : - the browser makes a connection and issues a first request - the webserver dedicates a child (or thread) to this connection, and passes it the first request - the child/thread responds to the first request, and then waits for more - the browser, in the response page, finds more links. Over the same TCP connection, it sends the next request - the same child/thread - which was waiting on that connection - receives the new request, and responds to it. Then it waits again for the next one. - etc.. - until at some point, the browser does not issue any additional requests on the connection. Then, *after the KeepAliveTimeout has expired*, the child/thread gives up, closesthe connection, and returns to the pool available for other requests from other browsers So the point is, if the KeepAliveTimeout is long (like 15 seconds), it means that a child/thread may be kept waiting, for nothing, up to that many seconds, although there is nothing coming anymore. I however wonder if this will fix the issue. The reason being, I haven't changed the website code at all the past few months and there hasn't been any increase in the website traffic too. Hence I am unable to understand why we are suddenly seeing an increase in the number of httpd processes. The only thing I changed is the session-timeout value from 30 minutes to 240 minutes. I guess that this is the Tomcat session timeout. That should have nothing to do with the above. I don't think that for Tomcat, a "session" is linked to a connection. It is more of a set of data saved somewhere, linked to the Tomcat session-id (the JSESSIONID cookie for instance). Tomcar retrieves it whenever a request comes in with the same session-id number. But it should not matter whether it is on the same TCP connection or not. What may be linked together however, is that one request to httpd results in one child/thread busy with it at the Apache httpd level. If that request is being forwarded to Tomcat by mod_jk, then it also holds onto one mod_jk/Tomcat connection. This connection then holds on to one thread in Tomcat, until the Tomcat thread (+webapp) has supplied the full response. All the while, this whole chain is unavailable for other requests. Thus, if there are many such requests under way, many Apache children/threads are busy, and Apache httpd will start additional ones (up to its limit) to service new requests that come in. So if for some reason, your Tomcat requests now take longer to be serviced, that should also, by retro-effect, increase the number of httpd children/threads being started. The bottleneck would be in Tomcat, but it would show up at the httpd level. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Thank you for the reply, Andre. I now understand how setting KeepAlive to On would improve the performance of a website (The Apache manual says that a 50% increase in throughput could be expected). So I changed the KeepAlive to On and restarted the server. I however wonder if this will fix the issue. The reason being, I haven't changed the website code at all the past few months and there hasn't been any increase in the website traffic too. Hence I am unable to understand why we are suddenly seeing an increase in the number of httpd processes. The only thing I changed is the session-timeout value from 30 minutes to 240 minutes. Thanks, Joe On Mon, Oct 5, 2009 at 1:04 PM, André Warnier wrote: > Joe Hansen wrote: >> >> Rainer, >> >> Here are the KeepAlive values in httpd.conf: >> >> KeepAlive Off >> MaxKeepAliveRequests 100 >> KeepAliveTimout 15 >> > Well, since you have "KeepAlive Off", the other 2 do not matter. > But as such, it means that each request of each browser is going to create a > new connection to the webserver, just for that one request. > So if there is a page with 10 links inside, you will end up > establishing (and tearing down) a total of 11 TCP connections (one for the > main page, one each for each ). > That may or may not have a bearing on the situation you are seeing. > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Joe Hansen wrote: Rainer, Here are the KeepAlive values in httpd.conf: KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimout 15 Well, since you have "KeepAlive Off", the other 2 do not matter. But as such, it means that each request of each browser is going to create a new connection to the webserver, just for that one request. So if there is a page with 10 links inside, you will end up establishing (and tearing down) a total of 11 TCP connections (one for the main page, one each for each ). That may or may not have a bearing on the situation you are seeing. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Rainer, Here are the KeepAlive values in httpd.conf: KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimout 15 Thanks, Joe > What are your KeepAlive* settings ? > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Thank you so much for your tips, Rainer! The websites went down yet again. Increasing the java heap size took care of the OutOfMemoryError, but the number of httpd processes keep increasing until the websites crash. I haven't added any new code in the past few months, hence I am surprised why the requests are getting stuck. Here's a link to the tomcat thread dumps: http://pastebin.com/m17eea139 Please let me know if you cannot view it and I will email the relevant portion of the catalina.out file to you. Is there an easy way to find out what code is causing the requests to get stuck? Thank you! Joe On Sun, Oct 4, 2009 at 2:36 PM, Rainer Jung wrote: > Hi Joe, > > On 04.10.2009 21:45, Joe Hansen wrote: >> Rainer, Thank you so much for your kind reply! >> >> I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am >> hoping that would fix the issue. I had configured our webserver to use >> Jasig's Central Authentication System (CAS). Recently I increased the >> session timeout from 30 minutes to 4 hours. I am guessing that must >> have had an impact on the number of tickets that the CAS could store >> in the Java's memory space. >> >> I did run the kill -QUIT command against the tomcat process. It did >> generate a huge output in the catalina.out file. I am unable to >> decipher it. I do not want to post it to the mailing list because its >> very long. Would you be able to please tell me what should I be >> looking for within this long thread dump? > > Can you put it somewhere on the web, so we can look at it, or are you > afraid there is something private in there? You could use pastebin or > something similar in case you do not have a public web server yourself. > > If you don't want to post in public, you can also mail it to me, I will > post the result, in case I find something relevant. > > Regards, > > Rainer > >> On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung wrote: >>> On 03.10.2009 20:07, Joe Hansen wrote: >>>> Hey All, >>>> >>>> I get this error (java.lang.OutOfMemoryError: Java heap space) after >>>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >>>> for a few hours. This problem started just since two days. Never had >>>> this issue before! >>>> >>>> I have also noticed that as soon as I startup the server, 9 httpd >>>> processes start. Number of httpd processes keep on increasing until I >>>> get the OutOfMemoryError. >>>> $ps -aef | grep httpd >>>> root 31984 1 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31987 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31988 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31989 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31990 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31991 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31992 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31993 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31994 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> >>> Sounds like requests get stuck or responses are only returned very slowly. >>> >>> I would take thread dumps during the time requests pile up (e.g. httpd >>> process count increases). Thread dumps are generated by "kil -QUIT" >>> against the Tomcat process. Result is written to catalina.out. Always >>> take afew thread dumps shortly after each other, e.g. 3 dumps each 3 >>> seconds apart from the previous one, so that you can find out, if a >>> status in a dump is pure coincidence or lasts for somewhat longer. >>> >>>> $ps -aef | grep tomcat >>>> root 31949 1 43 11:23 pts/0 00:00:58 /usr/java/jdk/bin/java >>>> -Djava.u >>>> il.logging.manager=org.apache.juli.ClassLoaderLogManager >>>> -Djava.util.logging.co >>>> fig.file=/usr/lib/apache-tomcat/conf/logging.properties >>>> -Djava.endorsed.dirs=/u >>>> r/lib/apache-tomcat/common/endorsed -classpath >>>> :/usr/lib/apache-tomcat/bin/boot >>>> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar >>>> -Dcatalina.base=/us >>>> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat >>>> -Djava.io.tmpdir=/usr >>>> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start >>> >>> There is no Java memory configuration included above (i.e. al defaults). >>> It might well be, that you have to explicitely set heap size, perm size >>> and if you like also eden and semi spaces. >>> >>>> Can someone on this list please help me resolve this issue. >>>> >>>> Thanks you, >>>> Joe >>> >>> Regards, >>> >>> Rainer >>> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Joe Hansen wrote: I found the following error message in the Apache logs: [Sat Oct 03 04:10:49 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting Here's a snippet from the httpd.conf, which deals with MaxClients. StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 I will watch out for the increase in the number of httpd processes. I am wondering if I should raise the MaxClients value in prefork.c and worker.c modules. Can anyone on this forum please explain why new httpd processes are spawned and why aren't the old processes terminated? What are your KeepAlive* settings ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Hi Joe, On 04.10.2009 21:45, Joe Hansen wrote: > Rainer, Thank you so much for your kind reply! > > I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am > hoping that would fix the issue. I had configured our webserver to use > Jasig's Central Authentication System (CAS). Recently I increased the > session timeout from 30 minutes to 4 hours. I am guessing that must > have had an impact on the number of tickets that the CAS could store > in the Java's memory space. > > I did run the kill -QUIT command against the tomcat process. It did > generate a huge output in the catalina.out file. I am unable to > decipher it. I do not want to post it to the mailing list because its > very long. Would you be able to please tell me what should I be > looking for within this long thread dump? Can you put it somewhere on the web, so we can look at it, or are you afraid there is something private in there? You could use pastebin or something similar in case you do not have a public web server yourself. If you don't want to post in public, you can also mail it to me, I will post the result, in case I find something relevant. Regards, Rainer > On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung wrote: >> On 03.10.2009 20:07, Joe Hansen wrote: >>> Hey All, >>> >>> I get this error (java.lang.OutOfMemoryError: Java heap space) after >>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >>> for a few hours. This problem started just since two days. Never had >>> this issue before! >>> >>> I have also noticed that as soon as I startup the server, 9 httpd >>> processes start. Number of httpd processes keep on increasing until I >>> get the OutOfMemoryError. >>> $ps -aef | grep httpd >>> root 31984 1 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31987 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31988 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31989 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31990 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31991 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31992 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31993 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> apache 31994 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >> >> Sounds like requests get stuck or responses are only returned very slowly. >> >> I would take thread dumps during the time requests pile up (e.g. httpd >> process count increases). Thread dumps are generated by "kil -QUIT" >> against the Tomcat process. Result is written to catalina.out. Always >> take afew thread dumps shortly after each other, e.g. 3 dumps each 3 >> seconds apart from the previous one, so that you can find out, if a >> status in a dump is pure coincidence or lasts for somewhat longer. >> >>> $ps -aef | grep tomcat >>> root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java >>> -Djava.u >>> il.logging.manager=org.apache.juli.ClassLoaderLogManager >>> -Djava.util.logging.co >>> fig.file=/usr/lib/apache-tomcat/conf/logging.properties >>> -Djava.endorsed.dirs=/u >>> r/lib/apache-tomcat/common/endorsed -classpath >>> :/usr/lib/apache-tomcat/bin/boot >>> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar >>> -Dcatalina.base=/us >>> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat >>> -Djava.io.tmpdir=/usr >>> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start >> >> There is no Java memory configuration included above (i.e. al defaults). >> It might well be, that you have to explicitely set heap size, perm size >> and if you like also eden and semi spaces. >> >>> Can someone on this list please help me resolve this issue. >>> >>> Thanks you, >>> Joe >> >> Regards, >> >> Rainer >> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
On 04.10.2009 22:17, Joe Hansen wrote: > I found the following error message in the Apache logs: > [Sat Oct 03 04:10:49 2009] [error] server reached MaxClients setting, > consider raising the MaxClients setting > > Here's a snippet from the httpd.conf, which deals with MaxClients. > > StartServers 8 > MinSpareServers5 > MaxSpareServers 20 > ServerLimit 256 > MaxClients 256 > MaxRequestsPerChild 4000 > > > > StartServers 2 > MaxClients 150 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadsPerChild 25 > MaxRequestsPerChild 0 > > > I will watch out for the increase in the number of httpd processes. I > am wondering if I should raise the MaxClients value in prefork.c and > worker.c modules. Can anyone on this forum please explain why new > httpd processes are spawned and why aren't the old processes > terminated? Likely because requests get stuck (no responses are returned), so the web server waits very long for Tomcat to produce a request and the stream of incoming new requests fill up all available threads in the web server. To find the reason, the thread dumps will be helpful. Regards, Rainer > On Sun, Oct 4, 2009 at 1:45 PM, Joe Hansen wrote: >> Rainer, Thank you so much for your kind reply! >> >> I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am >> hoping that would fix the issue. I had configured our webserver to use >> Jasig's Central Authentication System (CAS). Recently I increased the >> session timeout from 30 minutes to 4 hours. I am guessing that must >> have had an impact on the number of tickets that the CAS could store >> in the Java's memory space. >> >> I did run the kill -QUIT command against the tomcat process. It did >> generate a huge output in the catalina.out file. I am unable to >> decipher it. I do not want to post it to the mailing list because its >> very long. Would you be able to please tell me what should I be >> looking for within this long thread dump? >> >> Thanks again, Rainer :) >> >> Joe >> >> On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung wrote: >>> On 03.10.2009 20:07, Joe Hansen wrote: >>>> Hey All, >>>> >>>> I get this error (java.lang.OutOfMemoryError: Java heap space) after >>>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >>>> for a few hours. This problem started just since two days. Never had >>>> this issue before! >>>> >>>> I have also noticed that as soon as I startup the server, 9 httpd >>>> processes start. Number of httpd processes keep on increasing until I >>>> get the OutOfMemoryError. >>>> $ps -aef | grep httpd >>>> root 31984 1 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31987 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31988 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31989 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31990 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31991 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31992 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31993 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>>> apache 31994 31984 0 11:23 ?00:00:00 /usr/sbin/httpd >>> >>> Sounds like requests get stuck or responses are only returned very slowly. >>> >>> I would take thread dumps during the time requests pile up (e.g. httpd >>> process count increases). Thread dumps are generated by "kil -QUIT" >>> against the Tomcat process. Result is written to catalina.out. Always >>> take afew thread dumps shortly after each other, e.g. 3 dumps each 3 >>> seconds apart from the previous one, so that you can find out, if a >>> status in a dump is pure coincidence or lasts for somewhat longer. >>> >>>> $ps -aef | grep tomcat >>>> root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java >>>> -Djava.u >>>> il.logging.manager=org.apache.juli.ClassLoaderLogManager >>>> -Djava.util.logging.co >>>> fig.file=/usr/lib/apache-tomcat/conf/logging.properties >>>> -Djava.endorsed.dirs=/u >>>> r/lib/apache-tomcat/common/endorsed -classpath >>>> :/usr/lib/apache-tomcat/bin/boot >>>> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar >>>> -Dcatalina.base=/us >>>> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat >>>> -Djava.io.tmpdir=/usr >>>> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start >>> >>> There is no Java memory configuration included above (i.e. al defaults). >>> It might well be, that you have to explicitely set heap size, perm size >>> and if you like also eden and semi spaces. >>> >>>> Can someone on this list please help me resolve this issue. >>>> >>>> Thanks you, >>>> Joe >>> >>> Regards, >>> >>> Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
I found the following error message in the Apache logs: [Sat Oct 03 04:10:49 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting Here's a snippet from the httpd.conf, which deals with MaxClients. StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 I will watch out for the increase in the number of httpd processes. I am wondering if I should raise the MaxClients value in prefork.c and worker.c modules. Can anyone on this forum please explain why new httpd processes are spawned and why aren't the old processes terminated? Thanks, Joe On Sun, Oct 4, 2009 at 1:45 PM, Joe Hansen wrote: > Rainer, Thank you so much for your kind reply! > > I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am > hoping that would fix the issue. I had configured our webserver to use > Jasig's Central Authentication System (CAS). Recently I increased the > session timeout from 30 minutes to 4 hours. I am guessing that must > have had an impact on the number of tickets that the CAS could store > in the Java's memory space. > > I did run the kill -QUIT command against the tomcat process. It did > generate a huge output in the catalina.out file. I am unable to > decipher it. I do not want to post it to the mailing list because its > very long. Would you be able to please tell me what should I be > looking for within this long thread dump? > > Thanks again, Rainer :) > > Joe > > On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung wrote: >> On 03.10.2009 20:07, Joe Hansen wrote: >>> Hey All, >>> >>> I get this error (java.lang.OutOfMemoryError: Java heap space) after >>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >>> for a few hours. This problem started just since two days. Never had >>> this issue before! >>> >>> I have also noticed that as soon as I startup the server, 9 httpd >>> processes start. Number of httpd processes keep on increasing until I >>> get the OutOfMemoryError. >>> $ps -aef | grep httpd >>> root 31984 1 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31987 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31988 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31989 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31990 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31991 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31992 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31993 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> apache 31994 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> >> Sounds like requests get stuck or responses are only returned very slowly. >> >> I would take thread dumps during the time requests pile up (e.g. httpd >> process count increases). Thread dumps are generated by "kil -QUIT" >> against the Tomcat process. Result is written to catalina.out. Always >> take afew thread dumps shortly after each other, e.g. 3 dumps each 3 >> seconds apart from the previous one, so that you can find out, if a >> status in a dump is pure coincidence or lasts for somewhat longer. >> >>> $ps -aef | grep tomcat >>> root 31949 1 43 11:23 pts/0 00:00:58 /usr/java/jdk/bin/java >>> -Djava.u >>> il.logging.manager=org.apache.juli.ClassLoaderLogManager >>> -Djava.util.logging.co >>> fig.file=/usr/lib/apache-tomcat/conf/logging.properties >>> -Djava.endorsed.dirs=/u >>> r/lib/apache-tomcat/common/endorsed -classpath >>> :/usr/lib/apache-tomcat/bin/boot >>> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar >>> -Dcatalina.base=/us >>> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat >>> -Djava.io.tmpdir=/usr >>> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start >> >> There is no Java memory configuration included above (i.e. al defaults). >> It might well be, that you have to explicitely set heap size, perm size >> and if you like also eden and semi spaces. >> >>> Can someone on this list please help me resolve this issue. >>> >>> Thanks you, >>> Joe >> >> Regards, >> >> Rainer >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
Rainer, Thank you so much for your kind reply! I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am hoping that would fix the issue. I had configured our webserver to use Jasig's Central Authentication System (CAS). Recently I increased the session timeout from 30 minutes to 4 hours. I am guessing that must have had an impact on the number of tickets that the CAS could store in the Java's memory space. I did run the kill -QUIT command against the tomcat process. It did generate a huge output in the catalina.out file. I am unable to decipher it. I do not want to post it to the mailing list because its very long. Would you be able to please tell me what should I be looking for within this long thread dump? Thanks again, Rainer :) Joe On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung wrote: > On 03.10.2009 20:07, Joe Hansen wrote: >> Hey All, >> >> I get this error (java.lang.OutOfMemoryError: Java heap space) after >> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >> for a few hours. This problem started just since two days. Never had >> this issue before! >> >> I have also noticed that as soon as I startup the server, 9 httpd >> processes start. Number of httpd processes keep on increasing until I >> get the OutOfMemoryError. >> $ps -aef | grep httpd >> root 31984 1 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31987 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31988 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31989 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31990 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31991 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31992 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31993 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >> apache 31994 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd > > Sounds like requests get stuck or responses are only returned very slowly. > > I would take thread dumps during the time requests pile up (e.g. httpd > process count increases). Thread dumps are generated by "kil -QUIT" > against the Tomcat process. Result is written to catalina.out. Always > take afew thread dumps shortly after each other, e.g. 3 dumps each 3 > seconds apart from the previous one, so that you can find out, if a > status in a dump is pure coincidence or lasts for somewhat longer. > >> $ps -aef | grep tomcat >> root 31949 1 43 11:23 pts/0 00:00:58 /usr/java/jdk/bin/java >> -Djava.u >> il.logging.manager=org.apache.juli.ClassLoaderLogManager >> -Djava.util.logging.co >> fig.file=/usr/lib/apache-tomcat/conf/logging.properties >> -Djava.endorsed.dirs=/u >> r/lib/apache-tomcat/common/endorsed -classpath >> :/usr/lib/apache-tomcat/bin/boot >> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar >> -Dcatalina.base=/us >> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat >> -Djava.io.tmpdir=/usr >> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start > > There is no Java memory configuration included above (i.e. al defaults). > It might well be, that you have to explicitely set heap size, perm size > and if you like also eden and semi spaces. > >> Can someone on this list please help me resolve this issue. >> >> Thanks you, >> Joe > > Regards, > > Rainer > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.OutOfMemoryError: Java heap space
On 03.10.2009 20:07, Joe Hansen wrote: > Hey All, > > I get this error (java.lang.OutOfMemoryError: Java heap space) after > my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running > for a few hours. This problem started just since two days. Never had > this issue before! > > I have also noticed that as soon as I startup the server, 9 httpd > processes start. Number of httpd processes keep on increasing until I > get the OutOfMemoryError. > $ps -aef | grep httpd > root 31984 1 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31987 31984 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31988 31984 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31989 31984 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31990 31984 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31991 31984 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31992 31984 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31993 31984 0 11:23 ?00:00:00 /usr/sbin/httpd > apache 31994 31984 0 11:23 ?00:00:00 /usr/sbin/httpd Sounds like requests get stuck or responses are only returned very slowly. I would take thread dumps during the time requests pile up (e.g. httpd process count increases). Thread dumps are generated by "kil -QUIT" against the Tomcat process. Result is written to catalina.out. Always take afew thread dumps shortly after each other, e.g. 3 dumps each 3 seconds apart from the previous one, so that you can find out, if a status in a dump is pure coincidence or lasts for somewhat longer. > $ps -aef | grep tomcat > root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java > -Djava.u > il.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.util.logging.co > fig.file=/usr/lib/apache-tomcat/conf/logging.properties > -Djava.endorsed.dirs=/u > r/lib/apache-tomcat/common/endorsed -classpath > :/usr/lib/apache-tomcat/bin/boot > trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar > -Dcatalina.base=/us > /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat > -Djava.io.tmpdir=/usr > lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start There is no Java memory configuration included above (i.e. al defaults). It might well be, that you have to explicitely set heap size, perm size and if you like also eden and semi spaces. > Can someone on this list please help me resolve this issue. > > Thanks you, > Joe Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
java.lang.OutOfMemoryError: Java heap space
Hey All, I get this error (java.lang.OutOfMemoryError: Java heap space) after my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running for a few hours. This problem started just since two days. Never had this issue before! I have also noticed that as soon as I startup the server, 9 httpd processes start. Number of httpd processes keep on increasing until I get the OutOfMemoryError. $ps -aef | grep httpd root 31984 1 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31987 31984 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31988 31984 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31989 31984 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31990 31984 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31991 31984 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31992 31984 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31993 31984 0 11:23 ?00:00:00 /usr/sbin/httpd apache 31994 31984 0 11:23 ?00:00:00 /usr/sbin/httpd $ps -aef | grep tomcat root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java -Djava.u il.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.co fig.file=/usr/lib/apache-tomcat/conf/logging.properties -Djava.endorsed.dirs=/u r/lib/apache-tomcat/common/endorsed -classpath :/usr/lib/apache-tomcat/bin/boot trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/us /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat -Djava.io.tmpdir=/usr lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start Can someone on this list please help me resolve this issue. Thanks you, Joe - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stefan, Stefan Rainer wrote: > we are running a Tomcat 5.0.28 on W2k Server with 2 GB RAM and JVM > 1.5.0_11-b03 which is mainly used as application server for (Axis-based) > SOAP Services. > > From time to time our Tomcat blows up to use almost the whole available > RAM and finally it "crashes". Hmm. Do your serialized objects have a lot of circular relationships? I'm not sure if Java serialization has gotten any better over the years, but I did a demo a few years ago with object graphs like: A -> B B -> A When serializing object A, B goes with it. Upon de-serialization, multiple A and B objects were recovered. If this foolishness still happens with Java serialization, and you have large objects (many MiB) being serialized via the session persistence mechanism, you could bust your heap trying to re-load them due to the behavior described above. I would try to trim-down the contents of your session, or mark some objects as transient or otherwise non-serializable. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkk1XzMACgkQ9CaO5/Lv0PCwBACfW9+JQMdpZdTcFVzqXNgzetde G5QAnija1IF98j+5GUxwi7CKPN9Q586P =Dv/d -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space
> From: Stefan Rainer [mailto:[EMAIL PROTECTED] > Subject: AW: IOException while loading persisted sessions / > java.lang.OutOfMemoryError: Java heap space > > Is there any possibiliy in Tomcat 5.0.28 to turn off > session-persistence? First, you should be aware that 5.0.28 is no longer supported. In 5.5 and more recent versions, look at the pathname attribute for the element: http://tomcat.apache.org/tomcat-5.5-doc/config/manager.html (I think that existed in 5.0 as well, but you'll need to check the doc that came with that version of Tomcat.) A element may be nested inside a ; if you place it inside the global element (in conf/context.xml in 5.5 and above), you can disable sessions persistence for all of your webapps with one configuration change. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: AW: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space
Stefan Rainer wrote: Hello Andre, thank you very much for your comments. We've modified the java heap memory configuration and will observe the system. Is there any possibiliy in Tomcat 5.0.28 to turn off session-persistence? I would like to avoid the creation of [catalina-home]\work\ . We don't use persistant sessions in our applications. I don't really know. But I think that [catalina-home]\work\ may contain more things than just session storage (compiled classes maybe), so you may not really be able to get rid of it as such. If you want to monitor your Tomcat's memory usage more finely, the "jconsole" program (part of the Java JDK) provides a nice graphical way to examine the memory usage in real-time. Information here : http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jconsole.html ?? Thanks, Stefan -Ursprüngliche Nachricht- Von: André Warnier [mailto:[EMAIL PROTECTED] Gesendet: Montag, 01. Dezember 2008 11:47 An: Tomcat Users List Betreff: Re: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space Sorry, I was so shocked by the last info about 64 MB, that I missed what you wrote about the physical machine... Stefan Rainer wrote: Hello, we are running a Tomcat 5.0.28 on W2k Server with 2 GB RAM and JVM 1.5.0_11-b03 which is mainly used as application server for (Axis-based) SOAP Services. So, the machine itself has enough memory, it's just that the way Tomcat is started, the JVM under which it runs does not receive more than 64 MB of Heap space to run, which is really low for a server. The fix is probably simple ; you need to increase how much heap memory Java can play with. Since you are running this under Windows, I presume it runs as a service, and probably in your Tomcat/bin directory you have 2 files : tomcat5.exe and tomcat5w.exe. tomcat5w.exe is a GUI application, which allows you to set the parameters under which the JVM of Tomcat is started. Double-click on it, select the "Java" tab. At the bottom of that tab, there are 2 input boxes : Initial Memory Pool and Maximum Memory Pool. Enter 256 in each of them, and click OK. Then restart your Tomcat service and see if you still get the same problem. These parameters are the equivalent of specifying the "-Xms256M and -Xmx256M" on the java command-line that starts Tomcat. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
AW: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space
Hello Andre, thank you very much for your comments. We've modified the java heap memory configuration and will observe the system. Is there any possibiliy in Tomcat 5.0.28 to turn off session-persistence? I would like to avoid the creation of [catalina-home]\work\ . We don't use persistant sessions in our applications. ?? Thanks, Stefan -Ursprüngliche Nachricht- Von: André Warnier [mailto:[EMAIL PROTECTED] Gesendet: Montag, 01. Dezember 2008 11:47 An: Tomcat Users List Betreff: Re: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space Sorry, I was so shocked by the last info about 64 MB, that I missed what you wrote about the physical machine... Stefan Rainer wrote: > Hello, > > we are running a Tomcat 5.0.28 on W2k Server with 2 GB RAM and JVM > 1.5.0_11-b03 which is mainly used as application server for (Axis-based) > SOAP Services. > So, the machine itself has enough memory, it's just that the way Tomcat is started, the JVM under which it runs does not receive more than 64 MB of Heap space to run, which is really low for a server. The fix is probably simple ; you need to increase how much heap memory Java can play with. Since you are running this under Windows, I presume it runs as a service, and probably in your Tomcat/bin directory you have 2 files : tomcat5.exe and tomcat5w.exe. tomcat5w.exe is a GUI application, which allows you to set the parameters under which the JVM of Tomcat is started. Double-click on it, select the "Java" tab. At the bottom of that tab, there are 2 input boxes : Initial Memory Pool and Maximum Memory Pool. Enter 256 in each of them, and click OK. Then restart your Tomcat service and see if you still get the same problem. These parameters are the equivalent of specifying the "-Xms256M and -Xmx256M" on the java command-line that starts Tomcat. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space
Sorry, I was so shocked by the last info about 64 MB, that I missed what you wrote about the physical machine... Stefan Rainer wrote: Hello, we are running a Tomcat 5.0.28 on W2k Server with 2 GB RAM and JVM 1.5.0_11-b03 which is mainly used as application server for (Axis-based) SOAP Services. So, the machine itself has enough memory, it's just that the way Tomcat is started, the JVM under which it runs does not receive more than 64 MB of Heap space to run, which is really low for a server. The fix is probably simple ; you need to increase how much heap memory Java can play with. Since you are running this under Windows, I presume it runs as a service, and probably in your Tomcat/bin directory you have 2 files : tomcat5.exe and tomcat5w.exe. tomcat5w.exe is a GUI application, which allows you to set the parameters under which the JVM of Tomcat is started. Double-click on it, select the "Java" tab. At the bottom of that tab, there are 2 input boxes : Initial Memory Pool and Maximum Memory Pool. Enter 256 in each of them, and click OK. Then restart your Tomcat service and see if you still get the same problem. These parameters are the equivalent of specifying the "-Xms256M and -Xmx256M" on the java command-line that starts Tomcat. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space
Stefan Rainer wrote: [...] SEVERE: An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space Exception in thread "http-16302-Processor151" java.lang.OutOfMemoryError: Java heap space Extract from configuration / status, if needed: Free memory: 10.42 MB Total memory: 63.56 MB Max memory: 63.56 MB Hi. I am far from the expert, but according to some recent answers to some recent questions of my own, what I see above would lead me to say (without disrespect) : are you kidding ? In other words, are you really running this Tomcat on a machine that has only 64 MB of RAM ? or starting Tomcat under a JVM that can only use 64 MB of memory ? That is very, very little. To explain : I am running a Tomcat in an in-house test machine with a total of 512 MB ram, and starting Tomcat with 200Mb of Heap space (-Xms 200M -Xmx200M), and I have been told here to get a serious machine.. The good news is that for the mere 50$ that it would cost you to add another 1Gb of Ram to this machine, you would probably solve your problem. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
IOException while loading persisted sessions / java.lang.OutOfMemoryError: Java heap space
Hello, we are running a Tomcat 5.0.28 on W2k Server with 2 GB RAM and JVM 1.5.0_11-b03 which is mainly used as application server for (Axis-based) SOAP Services. From time to time our Tomcat blows up to use almost the whole available RAM and finally it "crashes". In the logfile, we find the following: - IOException while loading persisted sessions: java.io.EOFException java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source) at java.io.ObjectInputStream.readStreamHeader(Unknown Source) at java.io.ObjectInputStream.(Unknown Source) at org.apache.catalina.util.CustomObjectInputStream.(CustomObjectInputStr eam.java:56) [] - Exception loading sessions from persistent storage java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source) at java.io.ObjectInputStream.readStreamHeader(Unknown Source) at java.io.ObjectInputStream.(Unknown Source) [] After that a restart of Tomcat is not possible and the following lines can be found in stdout.log: SEVERE: An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space Exception in thread "http-16302-Processor151" java.lang.OutOfMemoryError: Java heap space 28.11.2008 20:08:47 org.apache.coyote.http11.Http11Processor process SEVERE: Error processing request java.lang.OutOfMemoryError: Java heap space Only a reboot of the whole server fixes the problem and tomcat works again as exptected. I have read in some forum, that we could clean the ${catalina.home}/work/Catalina/localhost/ directory to enable a tomcat restart without server restart. We haven't been able to try that yet, but that fixes only the symptomatic not the problem. Does anyone have an idea/experience where the problem comes from? Something to do with memory / session management? Extract from configuration / status, if needed: Free memory: 10.42 MB Total memory: 63.56 MB Max memory: 63.56 MB Max threads: 150 Min spare threads: 25 Max spare threads: 75 Current thread count: 84 Current thread busy: 13 Max processing time: 876219 ms Processing time: 153801 s Request count: 283471 Error count: 182 Bytes received: 552.95 MB Bytes sent: 549.65 MB Many thanks in advance, Stefan - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: java.lang.OutOfMemoryError: Java heap space
Add -XX:+HeapDumpOnOutOfMemoryError to your command line options, this will create a .hprof file when you get an OOM error, then simply analyze this file with a little tool called YourKit (www.yourkit.com) excellent profiler Filip Jean-Sebastien Pilon wrote: Hello, I have been experiencing issues with a Tomcat server. Everyday we get the following error: java.lang.OutOfMemoryError: Java heap space This machine runs tomcat with 2 webapps and some java based cronjobs. I start tomcat with the following java options: -Xms128m -Xmx512m Tomcat Version Apache Tomcat/5.0.28 JVM Version 1.5.0_10-b03 What could be my problem, where should I look next ? Thanks. NOTICE: This email contains privileged and confidential information and is intended only for the individual to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this transmission by mistake and delete this communication from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. AVIS: Le présent courriel contient des renseignements de nature privilégiée et confidentielle et n’est destiné qu'à la personne à qui il est adressé. Si vous n’êtes pas le destinataire prévu, vous êtes par les présentes avisés que toute diffusion, distribution ou reproduction de cette communication est strictement interdite. Si vous avez reçu ce courriel par erreur, veuillez en aviser immédiatement l’expéditeur et le supprimer de votre système. Notez que la transmission de courriel ne peut en aucun cas être considéré comme inviolable ou exempt d’erreur puisque les informations qu’il contient pourraient être interceptés, corrompues, perdues, détruites, arrivées en retard ou incomplètes ou contenir un virus. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: java.lang.OutOfMemoryError: Java heap space
> From: Jean-Sebastien Pilon [mailto:[EMAIL PROTECTED] > Subject: java.lang.OutOfMemoryError: Java heap space > > What could be my problem Your application has a memory leak (or you might be running out of PermGen space). > where should I look next ? http://tomcat.apache.org/faq/memory.html - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
java.lang.OutOfMemoryError: Java heap space
Hello, I have been experiencing issues with a Tomcat server. Everyday we get the following error: java.lang.OutOfMemoryError: Java heap space This machine runs tomcat with 2 webapps and some java based cronjobs. I start tomcat with the following java options: -Xms128m -Xmx512m Tomcat Version Apache Tomcat/5.0.28 JVM Version 1.5.0_10-b03 What could be my problem, where should I look next ? Thanks. NOTICE: This email contains privileged and confidential information and is intended only for the individual to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this transmission by mistake and delete this communication from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. AVIS: Le présent courriel contient des renseignements de nature privilégiée et confidentielle et nest destiné qu'à la personne à qui il est adressé. Si vous nêtes pas le destinataire prévu, vous êtes par les présentes avisés que toute diffusion, distribution ou reproduction de cette communication est strictement interdite. Si vous avez reçu ce courriel par erreur, veuillez en aviser immédiatement lexpéditeur et le supprimer de votre système. Notez que la transmission de courriel ne peut en aucun cas être considéré comme inviolable ou exempt derreur puisque les informations quil contient pourraient être interceptés, corrompues, perdues, détruites, arrivées en retard ou incomplètes ou contenir un virus. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]