Tomcat Tuning Memory leak
Hi All I am working on Tomcat 4.1.24 on Solatis-8, 12 CPU, 24GB RAM Machine. I am using Apache 2.0.43 and the Jdk version is 1.4.1_02. When I start my server then after 5-6 hours my server becomes very slow and then I need to restart my server. I am taking the top stats for the tomcat and have found that normally the thread remains between 100- 140 but when the system crashes it reaches to 372 and the memory gradually increases from 256 to some where around 660. The top stats are as follows PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND 13082 root 115 00 653M 578M cpu/290 313:31 37.64% java 13082 root 139 70 654M 579M cpu/291 388:08 40.78% java 13082 root 143 00 654M 580M cpu/419 466:39 41.43% java 13082 root 374 10 662M 589M cpu/323 547:47 47.03% java 13082 root 374 100 662M 589M cpu/291 625:42 53.59% java Here the Thread count increases from 143 to 374 in just 10 min. Even though the no of user accessing this site has decreased. I really dont know what the problem is... I have set the Catalina Option as "-Xms128M -Xmx512M" but even setting the value of -XMx1024 doesn't solved my problem. The connector settings in my server.xml are as follows Please tell me what is the problem with my configuration. Please help me Thanks & Regards Mohit - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Hi, I have the very same problem. I have tried everything possible with no outcome (the fork atribute for the jsp compiler did not help (with this I refer to a previous discussion here)). I suspect the CoyoteConnector being at fault but have no proof yet. I plan to run profiler but doing that with users hanging on our machine it requires a bit of courage. If you manage to solve this problem somehow please let me know (on this mailing list) because there is at least one person _very_ interested in solving this problem. Even if you succeed by trying another connector I would like to hear. Good luck and best regards, David Mohit Gupta wrote: Hi All I am working on Tomcat 4.1.24 on Solatis-8, 12 CPU, 24GB RAM Machine. I am using Apache 2.0.43 and the Jdk version is 1.4.1_02. When I start my server then after 5-6 hours my server becomes very slow and then I need to restart my server. I am taking the top stats for the tomcat and have found that normally the thread remains between 100- 140 but when the system crashes it reaches to 372 and the memory gradually increases from 256 to some where around 660. The top stats are as follows PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND 13082 root 115 00 653M 578M cpu/290 313:31 37.64% java 13082 root 139 70 654M 579M cpu/291 388:08 40.78% java 13082 root 143 00 654M 580M cpu/419 466:39 41.43% java 13082 root 374 10 662M 589M cpu/323 547:47 47.03% java 13082 root 374 100 662M 589M cpu/291 625:42 53.59% java Here the Thread count increases from 143 to 374 in just 10 min. Even though the no of user accessing this site has decreased. I really don’t know what the problem is... I have set the Catalina Option as "-Xms128M -Xmx512M" but even setting the value of -XMx1024 doesn't solved my problem. The connector settings in my server.xml are as follows Please tell me what is the problem with my configuration. Please help me Thanks & Regards Mohit - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Ooops. I was too fast with my previous post. My config is different form yours: Also my machine is a bit smaller ;-) (with linux). But the behaviour is very similar. Best, David David Strupl wrote: Hi, I have the very same problem. I have tried everything possible with no outcome (the fork atribute for the jsp compiler did not help (with this I refer to a previous discussion here)). I suspect the CoyoteConnector being at fault but have no proof yet. I plan to run profiler but doing that with users hanging on our machine it requires a bit of courage. If you manage to solve this problem somehow please let me know (on this mailing list) because there is at least one person _very_ interested in solving this problem. Even if you succeed by trying another connector I would like to hear. Good luck and best regards, David Mohit Gupta wrote: Hi All I am working on Tomcat 4.1.24 on Solatis-8, 12 CPU, 24GB RAM Machine. I am using Apache 2.0.43 and the Jdk version is 1.4.1_02. When I start my server then after 5-6 hours my server becomes very slow and then I need to restart my server. I am taking the top stats for the tomcat and have found that normally the thread remains between 100- 140 but when the system crashes it reaches to 372 and the memory gradually increases from 256 to some where around 660. The top stats are as follows PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND 13082 root 115 00 653M 578M cpu/290 313:31 37.64% java 13082 root 139 70 654M 579M cpu/291 388:08 40.78% java 13082 root 143 00 654M 580M cpu/419 466:39 41.43% java 13082 root 374 10 662M 589M cpu/323 547:47 47.03% java 13082 root 374 100 662M 589M cpu/291 625:42 53.59% java Here the Thread count increases from 143 to 374 in just 10 min. Even though the no of user accessing this site has decreased. I really don’t know what the problem is... I have set the Catalina Option as "-Xms128M -Xmx512M" but even setting the value of -XMx1024 doesn't solved my problem. The connector settings in my server.xml are as follows Please tell me what is the problem with my configuration. Please help me Thanks & Regards Mohit - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Mohit Gupta wrote: I am working on Tomcat 4.1.24 on Solatis-8, 12 CPU, 24GB RAM Machine. I am using Apache 2.0.43 and the Jdk version is 1.4.1_02. When I start my server then after 5-6 hours my server becomes very slow and then I need to restart my server. I am taking the top stats for the tomcat and have found that normally the thread remains between 100- 140 but when the system crashes it reaches to 372 and the memory gradually increases from 256 to some where around 660. The top stats are as follows PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND 13082 root 115 00 653M 578M cpu/290 313:31 37.64% java 13082 root 139 70 654M 579M cpu/291 388:08 40.78% java 13082 root 143 00 654M 580M cpu/419 466:39 41.43% java 13082 root 374 10 662M 589M cpu/323 547:47 47.03% java 13082 root 374 100 662M 589M cpu/291 625:42 53.59% java Here the Thread count increases from 143 to 374 in just 10 min. Even though the no of user accessing this site has decreased. I really don’t know what the problem is... I have set the Catalina Option as "-Xms128M -Xmx512M" but even setting the value of -XMx1024 doesn't solved my problem. The connector settings in my server.xml are as follows Please tell me what is the problem with my configuration. Please help me First thing to find out is to figure out what all the threads are doing. Send the Tomcat process a -QUIT signal (`kill -QUIT 13082`) and capture the output which will be sent to stdout. You will get a stack trace for each thread showing what it is doing. You will probably find that the threads are busy somewhere in your application code. Additionally, upgrading to the latest Tomcat (4.1.27 or 5.0.16) and JDK (1.4.2_03) is a good idea as the latest versions have bug fixes and performance improvements. I doubt it is a Tomcat issue, it is more than likely an issue with your application, but the stack trace will show the cause. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
David Rees wrote: Additionally, upgrading to the latest Tomcat (4.1.27 or 5.0.16) and JDK (1.4.2_03) is a good idea as the latest versions have bug fixes and performance improvements. I doubt it is a Tomcat issue, it is more than likely an issue with your application, but the stack trace will show the cause. I have tomcat 4.1.29, JDK1.4.2. Also my app was completely ok with tomcat 3.x.x. The 100% of processor time occurs in my case only after the OOME. Before the OOME all the threads (both tomcat's and mine) are happy. Thanks for any hints, David - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Hi! David Strupl wrote: I have tomcat 4.1.29, JDK1.4.2. Also my app was completely ok with tomcat 3.x.x. The 100% of processor time occurs in my case only after the OOME. Before the OOME all the threads (both tomcat's and mine) are happy. The processor usage is not too surprising. When your machine runs out of memory and there are still busy processes, most of the cpu time will go into swapping in and out those processes. Still, since most prople are perfectly happy with the tomcat build you are using, there seems to be a problem with your application code. Perhaps you could elaborate a bit further here? And, as we had that subject a few days ago, are you by chance re-using StringBuffer instances ;-) Regards, Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Philipp Taprogge wrote: The processor usage is not too surprising. When your machine runs out of memory and there are still busy processes, most of the cpu time will go into swapping in and out those processes. Still, since most prople are perfectly happy with the tomcat build you are using, there seems to be a problem with your application code. Perhaps you could elaborate a bit further here? And, as we had that subject a few days ago, are you by chance re-using StringBuffer instances ;-) Yes - I was also not surprised with the CPU after the OOME. I will try to prepare a heap shapshot and post it here (only a link, don't worry) after I manage to get a usefull one. My application does not do anything special - it is an online gaming server with couple of JSPs and servlets. The profiling has not revealed any memory leak in my code so far. Also the code was running for several years in tomcat 3.x without any problem. After migrating to tomcat 4.x the nightmare began. Thanks for any hints, David - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Hi again! David Strupl wrote: Also the code was running for several years in tomcat 3.x without any problem. After migrating to tomcat 4.x the nightmare began. Have you changed the JRE as well or are you running the tomcat 4 instance in the same VM as the tomcat 3 before? Just to make sure your problems are not arising from changes made to the JVM in the meantime. Regards, Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Philipp Taprogge wrote: Have you changed the JRE as well or are you running the tomcat 4 instance in the same VM as the tomcat 3 before? Just to make sure your problems are not arising from changes made to the JVM in the meantime. I have upgraged the VM as well as the OS on the machine. In fact the whole machine is different now (being it dual-pentium this time). Right now I am further upgrading SUN JDK 1.4.2 to SUN 1.4.2_03 if that is going to have any impact. David - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat Tuning Memory leak
Howdy, >Yes - I was also not surprised with the CPU after the OOME. I will try >to prepare a heap shapshot and post it here (only a link, don't worry) >after I manage to get a usefull one. Make sure it's one before the OutOfMemoryError ;) >server with couple of JSPs and servlets. The profiling has not revealed >any memory leak in my code so far. Also the code was running for several >years in tomcat 3.x without any problem. After migrating to tomcat 4.x >the nightmare began. There's no doubt tomcat 4 (and 5's) memory usage characteristics are significantly different from tomcat 3. Nearly everything in the code is significantly different. Is it possible your app doesn't have any leaks, but simply needs more memory under load than you're allocating it? (BTW, to refresh my memory, is this setup where the JSP pages change hourly?) Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Shapira, Yoav wrote: (BTW, to refresh my memory, is this setup where the JSP pages change hourly?) I have already changed this ;-) Also added the fork attribute to true for jsp compile. It is not caused by the app - after the app starts and first 100 or so users connect the memory jumps up to approx 130 MB. But during the next 24 hours it eats more than 300 megs. From what I have seen from the profiler so far number of my objects is fairly decent. But there are more and more approx 100k (98320 chars) buffers created somewhere. I will post more precise info later after I a. upgrade to the latest and greatest JDK 1.4.2_03 b. create some HTML output of the heap Thanks for any hints so far, best regards, David - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
David Strupl wrote: I have already changed this ;-) Also added the fork attribute to true for jsp compile. It is not caused by the app - after the app starts and first 100 or so users connect the memory jumps up to approx 130 MB. But during the next 24 hours it eats more than 300 megs. From what I have seen from the profiler so far number of my objects is fairly decent. But there are more and more approx 100k (98320 chars) buffers created somewhere. I will post more precise info later after I a. upgrade to the latest and greatest JDK 1.4.2_03 b. create some HTML output of the heap Your JSP page shouldn't have any business logic in it, that should be in beans that your JSP page uses. You can test the beans in a profiler and see if there is any memory leaks. If you are using JDK1.4.1 then that will probably be your problem as there was a memory leak in the String class that was fixed in JDK1.4.2. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Hi (yes that's me again), I have more details for those that would be willing to help. I have started to use IBM's JDK that has the nice memory dump feature set up so that whenever OutOfMemory occurs the heap is dumped. So my heap is here http://hry.atlas.cz/zaloha/heapdump.20040114.160700.8886.txt.gz Probably no need to download the 12MB - I have tried to process it and interesting result is this: I have used the k function of HeapRoots205 (here is its description): "This executes by first running 'p 0x1234'. This leaves every object reachable by 0x1234 owned by it. Then any objects which are owned by 0x1234 but reachable from outside of this set are removed. This leaves 0x1234 owning exactly those objects which are only reachable thru' it." on following object: > k 0x10102818 Calculating keep-alive size of 0x10102818 'org/apache/coyote/RequestGroupInfo' ... Requesting 17 mb of heapspace to process heapdump ... done. Finding pure Roots .. done. DFS from 0x10102818 . done. Total reach (inclusive) 1,245,481 objects. Total-size is 299,128,896. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
My previous post should be longer ... here is the rest: DFS from pure Roots ...done. DFS from objects unreached from Roots ...done. Found 1,051,382 objects which are reachable from elsewhere. These will be taken from 0x10102818 and then owned by other objects ... DFS from 0x10102818 . done. Keep-alive size (inclusive) is 194,099 objects. Total-size is 154,016,544. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Still I don't see the whole thing - and trying to send the rest here: If I interpret it correctly the instance of org/apache/coyote/RequestGroupInfo holds 150 MB of heap memory. Also if I understand it correctly from here http://cvs.apache.org/viewcvs.cgi/jakarta-tomcat-connectors/coyote/src/java/org/apache/coyote/ someone just recently tried to do something with it. Am I totally paranoic when I suspect that the memory leak is in tomcat? Thanks for any hints (as usually) and best regards, David - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat Tuning Memory leak
Howdy, >If I interpret it correctly the instance of >org/apache/coyote/RequestGroupInfo >holds 150 MB of heap memory. Also if I understand it correctly from here >http://cvs.apache.org/viewcvs.cgi/jakarta-tomcat- >connectors/coyote/src/java/org/apache/coyote/ >someone just recently tried to do something with it. > >Am I totally paranoic when I suspect that the memory leak is in tomcat? You're not paranoid. There's a memory leak related to the RequestGroup/RequestGroupInfo connector code. It's been discussed during the past week on the dev mailing list, and addressed within the past couple of days. You can try the tomcat 5.0.18 build which has the fix. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Tuning Memory leak
Shapira, Yoav wrote: You're not paranoid. There's a memory leak related to the RequestGroup/RequestGroupInfo connector code. It's been discussed during the past week on the dev mailing list, and addressed within the past couple of days. You can try the tomcat 5.0.18 build which has the fix. Yoav Shapira Thanks for the pointer. Although they talk only about 5.0.x I thing that the problem is present even in 4.1.29 - it might make sense to only downgrade to 4.1.27 because the classes in question were not there ... I will first try to set minProcessors==maxProcessors and see whether it helps even in 4.1.29. If it won't help I will think about moving to some other revision. Thanks again and best regards, David - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
