RE: OutOfMemoryError with Seemingly Plenty of Memory
Do you edit the web.xml of your web application? That will cause Tomcat to redeploy the application. We had an application that would do this during their installation (so, only one time really), but I can imagine an application that might be trying to dynamically deploy servlets and set their mappings in the web.xml. I'm guessing this is NOT your situation though, since you'd probably already be thinking along the lines of redeploys. >>> [EMAIL PROTECTED] 1/27/04 10:52:34 AM >>> Hmmm...we don't do hot deploys, but the different generations are definitely something to look at. Thanks! Dhruva --- Jeff Tulley <[EMAIL PROTECTED]> wrote: > A while back somebody mentioned that if you continuously redeploy a > web > application, the symptoms you are mentioning occur. This is because > of > all of the objects that are stored in the "permanent generation" > memory. > This permanent generation runs out of space, even though the new > generation and old generation heaps still have room to grow. (The > permanent generation's size is a fixed percentage of the total > memory). > I do not know if this is something that can or is fixed in the most > recent builds and latest major version (5) of Tomcat. > > On NetWare, we have a view into the memory where we can see how big > the > perm gen is and what percentage of it is allocated. I do not know if > such a thing exists on other platforms though. If it does, I > certainly > have not yet found it. > > Searching the archives (at http://marc.theaimsgroup.com ), it looks > like it could have something to do with the classloader layout of > your > application: > > http://marc.theaimsgroup.com/?l=tomcat-user&m=107046034505454&w=2 > > > >Although the possibility of a memory leak (sorry--unintended object > >retention) has not been completely ruled out (yes, I'm running the > >applications through a profiler), I'm quite puzzled that this > problem > >would occur when, by all indications, memory use is nowhere near the > >limit. Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Wasn't there also a thread recently about a leak that can be configured away? The usage pattern to invoke the 'leak' was to load the site, then leave it alone, then load it again. It involved the worker thread pool for connections and JMX registration of said threads? The pool thought it was downsizing itself, but in fact it was leaking them away. So if you have lots of up and down loading going on, perhaps you can look into this further. > -Original Message- > From: Jeff Tulley [mailto:[EMAIL PROTECTED] > Sent: Tuesday, January 27, 2004 11:00 AM > To: [EMAIL PROTECTED] > Subject: RE: OutOfMemoryError with Seemingly Plenty of Memory > > > A while back somebody mentioned that if you continuously > redeploy a web application, the symptoms you are mentioning > occur. This is because of all of the objects that are stored > in the "permanent generation" memory. This permanent > generation runs out of space, even though the new generation > and old generation heaps still have room to grow. (The > permanent generation's size is a fixed percentage of the > total memory). > I do not know if this is something that can or is fixed in > the most recent builds and latest major version (5) of Tomcat. > > On NetWare, we have a view into the memory where we can see > how big the perm gen is and what percentage of it is > allocated. I do not know if such a thing exists on other > platforms though. If it does, I certainly have not yet found it. > > Searching the archives (at http://marc.theaimsgroup.com ), it > looks like it could have something to do with the classloader > layout of your > application: > http://marc.theaimsgroup.com/?l=tomcat-user&m=107046034505454&w=2 >Although the possibility of a memory leak (sorry--unintended object >retention) has not been completely ruled out (yes, I'm running the >applications through a profiler), I'm quite puzzled that this problem >would occur when, by all indications, memory use is nowhere near the >limit. Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Hmmm...we don't do hot deploys, but the different generations are definitely something to look at. Thanks! Dhruva --- Jeff Tulley <[EMAIL PROTECTED]> wrote: > A while back somebody mentioned that if you continuously redeploy a > web > application, the symptoms you are mentioning occur. This is because > of > all of the objects that are stored in the "permanent generation" > memory. > This permanent generation runs out of space, even though the new > generation and old generation heaps still have room to grow. (The > permanent generation's size is a fixed percentage of the total > memory). > I do not know if this is something that can or is fixed in the most > recent builds and latest major version (5) of Tomcat. > > On NetWare, we have a view into the memory where we can see how big > the > perm gen is and what percentage of it is allocated. I do not know if > such a thing exists on other platforms though. If it does, I > certainly > have not yet found it. > > Searching the archives (at http://marc.theaimsgroup.com ), it looks > like it could have something to do with the classloader layout of > your > application: > > http://marc.theaimsgroup.com/?l=tomcat-user&m=107046034505454&w=2 > > > >Although the possibility of a memory leak (sorry--unintended object > >retention) has not been completely ruled out (yes, I'm running the > >applications through a profiler), I'm quite puzzled that this > problem > >would occur when, by all indications, memory use is nowhere near the > >limit. > > > Jeff Tulley ([EMAIL PROTECTED]) > (801)861-5322 > Novell, Inc., The Leading Provider of Net Business Solutions > http://www.novell.com > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > __ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
A while back somebody mentioned that if you continuously redeploy a web application, the symptoms you are mentioning occur. This is because of all of the objects that are stored in the "permanent generation" memory. This permanent generation runs out of space, even though the new generation and old generation heaps still have room to grow. (The permanent generation's size is a fixed percentage of the total memory). I do not know if this is something that can or is fixed in the most recent builds and latest major version (5) of Tomcat. On NetWare, we have a view into the memory where we can see how big the perm gen is and what percentage of it is allocated. I do not know if such a thing exists on other platforms though. If it does, I certainly have not yet found it. Searching the archives (at http://marc.theaimsgroup.com ), it looks like it could have something to do with the classloader layout of your application: http://marc.theaimsgroup.com/?l=tomcat-user&m=107046034505454&w=2 >Although the possibility of a memory leak (sorry--unintended object >retention) has not been completely ruled out (yes, I'm running the >applications through a profiler), I'm quite puzzled that this problem >would occur when, by all indications, memory use is nowhere near the >limit. Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Well, I have made an assumption in most cases. The OutOfMemoryError is logged with no timestamp to $CATALINA_HOME/logs/catalina.out. We have a script that periodically (every 15 minutes) hits the applications and notifies us of any error response (either status above 400 or the word "ERROR" in the content). When I get such a notification, I immediately check the log file (catalina.out) for the OutOfMemoryError, and it is always near or at the bottom of the file. I know that in and of itself is no indication of when the error actually occurs. The applications themselves are set up to use syslog (through Log4J). Sometimes the OutOfMemoryError is logged here with a timestamp, and always occurs within minutes of my notification. --- "Shapira, Yoav" <[EMAIL PROTECTED]> wrote: > > Howdy, > > >Although the possibility of a memory leak (sorry--unintended object > >retention) has not been completely ruled out (yes, I'm running the > >applications through a profiler), I'm quite puzzled that this > problem > >would occur when, by all indications, memory use is nowhere near the > >limit. > > How do you know when the OutOfMemoryError occurred? > > Yoav Shapira > > > > This e-mail, including any attachments, is a confidential business > communication, and may contain information that is confidential, > proprietary and/or privileged. This e-mail is intended only for the > individual(s) to whom it is addressed, and may not be saved, copied, > printed, disclosed or used by anyone else. If you are not the(an) > intended recipient, please immediately delete this e-mail from your > computer system and notify the sender. Thank you. > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > __ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Howdy, >Although the possibility of a memory leak (sorry--unintended object >retention) has not been completely ruled out (yes, I'm running the >applications through a profiler), I'm quite puzzled that this problem >would occur when, by all indications, memory use is nowhere near the >limit. How do you know when the OutOfMemoryError occurred? Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
OutOfMemoryError with Seemingly Plenty of Memory
I am troubleshooting an issue we've been having with Tomcat 4.1.27. We have several websites on one instance, and it regularly throws an OutOfMemoryError, requiring a restart of Tomcat. Some observations (warning--this is a bit long): * The box runs two 2.8GHz Pentium 4 Hyperthreaded Xeons, and has 2GB of RAM (recently upgraded from 1GB). * The OutOfMemoryError never contains a message (I believe an OutOfMemoryError is thrown with a message when the java process has used the maximum number of threads). * There is a single instance of Tomcat, housing 20 applications (including the ones that come with the standard distribution of Tomcat, examples, tomcat-docs, etc.--these apps see no traffic). * We have a servlet that is hit every five minutes. It calls Runtime.freeMemory(), Runtime.totalMemory() and Runtime.maxMemory(), after calling Runtime.gc(). This information is graphed (free, total - free, both vs. time). According to this graph, there is always plenty of memory available (I have seen the amount of free memory get quite low--below 1MB, but never when the problem occurs). * We're running Tomcat 4.1.27, Redhat Linux 7.3, Sun JDK 1.4.1_05. * Peak traffic is between 10:00 and 22:00 everyday. The heaviest day is Monday and the lightest Saturday. * Together, the applications service an average of 1600+ requests per hour. * Before the memory upgrade, we were running an initial heap size of 100MB, and max of 768MB. We are now running an initial heap size of 768MB and max of 1536MB. * Uptime used to be about 100 hours, but recently has fallen to 24 hours. This change does not coincide with any configuration change. I have not determined if there is a coincident change in traffic patterns. The memory upgrade has had no effect on uptime. * Before the memory upgrade, when the problem occurred I would find the java process spinning (i.e., processor use according to top was 99%), and it appeared to be using some swap space. Since the memory upgrade, no swap space has been used, and the spinning has not been observed. * At the time that the problem occurs, top reports that java is using 500-600MB of RAM. This has not changed since the memory upgrade. * Tomcat is set up not to refresh the context automatically. Although it is set up to refresh JSPs, we do not do this. No changes are ever made without restarting Tomcat. * We have the JSP compiler set up to fork (i.e., use another instance of the JVM). * The problem almost always occurs during hours of heavy use, although it has also been observed on Saturdays (the day of lightest use). * Diskspace use is nowhere near capacity. * For a few days, we ran Tomcat with verbose garbage collection (the "verbose:gc" flag). The problem seemed to occur just after a "full" garbage collection. Although the possibility of a memory leak (sorry--unintended object retention) has not been completely ruled out (yes, I'm running the applications through a profiler), I'm quite puzzled that this problem would occur when, by all indications, memory use is nowhere near the limit. Any insights or other things to look at would be greatly appreciated. Thanks, Dhruva __ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
