RE: Tips on tracking down memory leaks
> From: Greg McCane [mailto:gregmcc...@yahoo.ca] > Subject: Re: Tips on tracking down memory leaks > > Is there any danger in taking a heap dump on our system running in > production? Will it cause a significant performance hit or other nasty? No; taking a heap dump will cause a momentary blip (threads have to be suspended briefly), but you're likely not to even notice it. Running a heap profiler can impact performance, if you're CPU-limited. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tips on tracking down memory leaks
No. But historical results are not a guarantee for the future. It is very easy to make a heapdump with jmap and analyze it with MAT (http://www.eclipse.org/mat/) or other tools as suggested by others on this list. Ronald. Op woensdag, 13 januari 2010 11:44 schreef Greg McCane : Thanks Chuck. Is there any danger in taking a heap dump on our system running in production? Will it cause a significant performance hit or other nasty? Thanks again, Greg From: "Caldarale, Charles R" To: Tomcat Users List Sent: Tue, January 12, 2010 11:31:16 PM Subject: RE: Tips on tracking down memory leaks > From: Greg McCane [mailto:gregmcc...@yahoo.ca] > Subject: Tips on tracking down memory leaks > > The memory growth appears to be in large chunks rather > than slow, steady growth. Use a heap profiler to find out what's eating up the space and who is allocating it. Even the simple one (hprof) included in the 1.5 JDK will tell you that. Better ones (e.g., jhat) are available in 1.6, if you're willing to move up. There are also numerous 3rd-party profilers available, with YourKit being a favorite of many. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org __ The new Internet Explorer® 8 - Faster, safer, easier. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/
Re: Tips on tracking down memory leaks
Thanks a lot Joe, we are checking our code based on your suggestions. Cheers, Greg From: Joseph Morgan To: Tomcat Users List Sent: Tue, January 12, 2010 11:15:54 PM Subject: RE: Tips on tracking down memory leaks Greg, you've just awakened a 900 lb gorilla! My guess, based upon your description of the problem as large chunks at a time, is something related to reading and caching data, and this is why PMD and profiling tools won't help. Therefore, look at all the code accessing data from whatever storage you use (DB, File?), and find out where that data flows. I'd also guess is that someone is storing data in the context, maybe as a cache, or just because they don't know any better. So search for "setAttribute" calls in the code and trace the associated object. If that object points to the servlet context, then mark it as a candidate. This could be complicated by JSPs (if you are using them), because they could be storing data within the application scope within JSP pages (very bad practice, but very widely used). Also check out tag libraries, because it can be done there too. Check your session timeout to make sure it isn't too high (however, you did say it isn't slow growth, but check anyway). However, we could literally guess all day long, as there are dozens of ways code can eat up memory. Joe -Original Message- From: Greg McCane [mailto:gregmcc...@yahoo.ca] Sent: Monday, January 11, 2010 8:33 PM To: users@tomcat.apache.org Subject: Tips on tracking down memory leaks Hi Everyone, I am running tomcat 5.5 on Debian Linux (uname says Linux 2.6.26-2-amd64, /etc/debian_version says 5.0.2). The JVM version is 1.5.0_14-b03. We have 9 servlets running. The tomcat process itself is managed by monit. We see tomcat memory usage growing over time and have set monit to automatically restart it (using the /etc/init.d/tomcat5.5 startup script) when memory usage reaches 2.8 GB. It takes about 20 hours to get to this point. This is an attempt to avoid a tomcat crash/hang resulting in an outage for our users. The memory growth appears to be in large chunks rather than slow, steady growth. We have turned on incremental garbage collection with the following options: "-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:+CMSIncrementalDutyCycle=10 -XX:-TraceClassUnloading -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled" The JVM heap size is set with: "-Xmx3072M -Xms1024M " We have checked our code using pmd and have profiled it using TPT and Eclipse in our development environment. We have made improvements as a result of the findings from these tools, but we are still having problems. Any suggestions for next steps to help us get to the root cause of this problem? Thanks, Greg __ Looking for the perfect gift? Give the gift of Flickr! http://www.flickr.com/gift/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org __ Be smarter than spam. See how smart SpamGuard is at giving junk email the boot with the All-new Yahoo! Mail. Click on Options in Mail and switch to New Mail today or register for free at http://mail.yahoo.ca
Re: Tips on tracking down memory leaks
Thanks Chuck. Is there any danger in taking a heap dump on our system running in production? Will it cause a significant performance hit or other nasty? Thanks again, Greg From: "Caldarale, Charles R" To: Tomcat Users List Sent: Tue, January 12, 2010 11:31:16 PM Subject: RE: Tips on tracking down memory leaks > From: Greg McCane [mailto:gregmcc...@yahoo.ca] > Subject: Tips on tracking down memory leaks > > The memory growth appears to be in large chunks rather > than slow, steady growth. Use a heap profiler to find out what's eating up the space and who is allocating it. Even the simple one (hprof) included in the 1.5 JDK will tell you that. Better ones (e.g., jhat) are available in 1.6, if you're willing to move up. There are also numerous 3rd-party profilers available, with YourKit being a favorite of many. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org __ The new Internet Explorer® 8 - Faster, safer, easier. Optimized for Yahoo! Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/
RE: Tips on tracking down memory leaks
> From: Greg McCane [mailto:gregmcc...@yahoo.ca] > Subject: Tips on tracking down memory leaks > > The memory growth appears to be in large chunks rather > than slow, steady growth. Use a heap profiler to find out what's eating up the space and who is allocating it. Even the simple one (hprof) included in the 1.5 JDK will tell you that. Better ones (e.g., jhat) are available in 1.6, if you're willing to move up. There are also numerous 3rd-party profilers available, with YourKit being a favorite of many. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tips on tracking down memory leaks
Greg, you've just awakened a 900 lb gorilla! My guess, based upon your description of the problem as large chunks at a time, is something related to reading and caching data, and this is why PMD and profiling tools won't help. Therefore, look at all the code accessing data from whatever storage you use (DB, File?), and find out where that data flows. I'd also guess is that someone is storing data in the context, maybe as a cache, or just because they don't know any better. So search for "setAttribute" calls in the code and trace the associated object. If that object points to the servlet context, then mark it as a candidate. This could be complicated by JSPs (if you are using them), because they could be storing data within the application scope within JSP pages (very bad practice, but very widely used). Also check out tag libraries, because it can be done there too. Check your session timeout to make sure it isn't too high (however, you did say it isn't slow growth, but check anyway). However, we could literally guess all day long, as there are dozens of ways code can eat up memory. Joe -Original Message- From: Greg McCane [mailto:gregmcc...@yahoo.ca] Sent: Monday, January 11, 2010 8:33 PM To: users@tomcat.apache.org Subject: Tips on tracking down memory leaks Hi Everyone, I am running tomcat 5.5 on Debian Linux (uname says Linux 2.6.26-2-amd64, /etc/debian_version says 5.0.2). The JVM version is 1.5.0_14-b03. We have 9 servlets running. The tomcat process itself is managed by monit. We see tomcat memory usage growing over time and have set monit to automatically restart it (using the /etc/init.d/tomcat5.5 startup script) when memory usage reaches 2.8 GB. It takes about 20 hours to get to this point. This is an attempt to avoid a tomcat crash/hang resulting in an outage for our users. The memory growth appears to be in large chunks rather than slow, steady growth. We have turned on incremental garbage collection with the following options: "-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:+CMSIncrementalDutyCycle=10 -XX:-TraceClassUnloading -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled" The JVM heap size is set with: "-Xmx3072M -Xms1024M " We have checked our code using pmd and have profiled it using TPT and Eclipse in our development environment. We have made improvements as a result of the findings from these tools, but we are still having problems. Any suggestions for next steps to help us get to the root cause of this problem? Thanks, Greg __ Looking for the perfect gift? Give the gift of Flickr! http://www.flickr.com/gift/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tips on tracking down memory leaks
Hi Everyone, I am running tomcat 5.5 on Debian Linux (uname says Linux 2.6.26-2-amd64, /etc/debian_version says 5.0.2). The JVM version is 1.5.0_14-b03. We have 9 servlets running. The tomcat process itself is managed by monit. We see tomcat memory usage growing over time and have set monit to automatically restart it (using the /etc/init.d/tomcat5.5 startup script) when memory usage reaches 2.8 GB. It takes about 20 hours to get to this point. This is an attempt to avoid a tomcat crash/hang resulting in an outage for our users. The memory growth appears to be in large chunks rather than slow, steady growth. We have turned on incremental garbage collection with the following options: "-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:+CMSIncrementalDutyCycle=10 -XX:-TraceClassUnloading -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled" The JVM heap size is set with: "-Xmx3072M -Xms1024M " We have checked our code using pmd and have profiled it using TPT and Eclipse in our development environment. We have made improvements as a result of the findings from these tools, but we are still having problems. Any suggestions for next steps to help us get to the root cause of this problem? Thanks, Greg __ Looking for the perfect gift? Give the gift of Flickr! http://www.flickr.com/gift/
