RE: verbose gc
Check out Jeff Tulley's responses to my post OutOfMemoryError with Seemingly Plenty of Memory. You may also want to check out Sun's forums on the Java Virtual Machine. You will find a host of information on JVM tuning. As I said in my post, we would get the OutOfMemoryError about every 24 hours. I tried setting MaxPermSize to 256MB (haven't been able to determine the default value for Sun's JVM, but it's 64MB for HP's JVM). This sets the size of the permanent generation, into which longer-lived objects and class definitions are loaded (we have over 3500 classes over multiple web apps). I wasn't going to say anything to this list without being more certain, but Tomcat has been running with no problems since Friday afternoon (47 hours as of this writing)--the longest it's lasted all month. HTH, Dhruva --- Mohammed Javid - CTD, Chennai. [EMAIL PROTECTED] wrote: Hi, Thanks for all the help. We did loadtesting of our application running on tomcat 4.1.27 with jdk1.4 on solaris, we enabled the GC stack to get details of how GC works, we ran for 1 users with 5 users logging every 30 secs. We set min and max heap size as 512mb The test ran perfectly for first 2 hours and total heap ocupied was only 10448K out of total of 519168K , later as test case progessed for around 10 hours the tomcat stopped responding and it has thrown OutOfMemoryError , but as can be seen from GC output before outofmemory error the total heap occupied was only 30846K, though we have lot of more space on heap still tomcat crashed and give connectiontimeout for remaining pages. When we see the size occupied from top command , when tomcat is started it was 622Mb and at time of crash it was 723MB. We are looking for following clarifications 1 What could be the reason for tomcat to crash though we have lot of heap remaining. 2 Why the memory occupied as seend from top command increased though there was lot of space on heap 3 We noticed that though heap has space when size seen from top command reached 720 to 270 mb tomcat crashes. 4 How to make tomcat running without crash , certianly heap is not problem in this case 5 Is this a known issue, Any optimization to be done to avoid this problem. The below is the output of the GC after which we get outofmemory error [GC 55404.101: [DefNew Desired survivor size 2621440 bytes, new threshold 31 (max 31) - age 1: 359488 bytes, 359488 total - age 2: 314104 bytes, 673592 total : 167034K-657K(169600K), 0.2717012 secs] 197223K-30846K(519168K), 0.2745516 secs] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start(Native Method) at org.apache.catalina.connector.http.HttpProcessor.threadStart(HttpProcessor.j ava:1178) at org.apache.catalina.connector.http.HttpProcessor.start(HttpProcessor.java:12 62) at org.apache.catalina.connector.http.HttpConnector.newProcessor(HttpConnector. java:937) at org.apache.catalina.connector.http.HttpConnector.createProcessor(HttpConnect or.java:868) at org.apache.catalina.connector.http.HttpConnector.run(HttpConnector.java:1075 ) at java.lang.Thread.run(Thread.java:534) thanks and regards, Javid -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 10:00 PM To: '[EMAIL PROTECTED]' Subject:RE: verbose gc we are using tomcat 4.1 withjdk1.3, we have folowing clarifications 1) [Full GC 34521K-15504K(38328K), 0.5953540 secs] [GC 27533K-16335K(38328K), 0.0396336 secs] is the statements got in catalina.out by adding -server -verbose:gc in catalina.sh 2) what does the above statements got in catalina.out means, is garbage collection active. When i execute top command on solaris and see the memory it shows size and resisdent memory , what is the diference between the two. 3) I see that gc statements are logged to catalina.out but the memory usage keeps increasing, does it mean there is problem with garbage collection, 4) can we get more information like how many obejcts created / objects destroyed during a gc run, what options to pass to get that information thanks -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 2:18 PM To: '[EMAIL PROTECTED]' Subject:verbose gc Hi, When we give verbose:gc in cataline.sh during startup of tomcat where does the gc information get logged. I am not seeing any information in catalina.out. thanks. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http
RE: verbose gc
A good article : http://java.sun.com/docs/hotspot/gc1.4.2/ (Java's Sun site) Memory reserved by Tomcat via JVM is cut in three groups (more in reality...) : - Immortal - New object - Old Object Imaging you're reserved 512Mo for Tomcat. If you don't specify dispatch of the memory (at the Tomcat startup) you will be have the following (it's a funny example ^^ and certainly impossible in reality): - Immortal 64Mo - New object 224 Mo - Old Object 224 Mo If your application need to create a lot of immortals Tomcat will crash fastly than if you're are specifying more memory for immortal. Another funny dispatching : - Immortal 224Mo - New object 64 Mo - Old Object 224 Mo If your application need to create a lot of new objects, Tomcat will crash if garbace collector take his time to do his job. Regards, leakim -Original Message- From: Dhruva B. Reddy [mailto:[EMAIL PROTECTED] Sent: dimanche 1 février 2004 17:12 To: Tomcat Users List Subject: RE: verbose gc Check out Jeff Tulley's responses to my post OutOfMemoryError with Seemingly Plenty of Memory. You may also want to check out Sun's forums on the Java Virtual Machine. You will find a host of information on JVM tuning. As I said in my post, we would get the OutOfMemoryError about every 24 hours. I tried setting MaxPermSize to 256MB (haven't been able to determine the default value for Sun's JVM, but it's 64MB for HP's JVM). This sets the size of the permanent generation, into which longer-lived objects and class definitions are loaded (we have over 3500 classes over multiple web apps). I wasn't going to say anything to this list without being more certain, but Tomcat has been running with no problems since Friday afternoon (47 hours as of this writing)--the longest it's lasted all month. HTH, Dhruva --- Mohammed Javid - CTD, Chennai. [EMAIL PROTECTED] wrote: Hi, Thanks for all the help. We did loadtesting of our application running on tomcat 4.1.27 with jdk1.4 on solaris, we enabled the GC stack to get details of how GC works, we ran for 1 users with 5 users logging every 30 secs. We set min and max heap size as 512mb The test ran perfectly for first 2 hours and total heap ocupied was only 10448K out of total of 519168K , later as test case progessed for around 10 hours the tomcat stopped responding and it has thrown OutOfMemoryError , but as can be seen from GC output before outofmemory error the total heap occupied was only 30846K, though we have lot of more space on heap still tomcat crashed and give connectiontimeout for remaining pages. When we see the size occupied from top command , when tomcat is started it was 622Mb and at time of crash it was 723MB. We are looking for following clarifications 1 What could be the reason for tomcat to crash though we have lot of heap remaining. 2 Why the memory occupied as seend from top command increased though there was lot of space on heap 3 We noticed that though heap has space when size seen from top command reached 720 to 270 mb tomcat crashes. 4 How to make tomcat running without crash , certianly heap is not problem in this case 5 Is this a known issue, Any optimization to be done to avoid this problem. The below is the output of the GC after which we get outofmemory error [GC 55404.101: [DefNew Desired survivor size 2621440 bytes, new threshold 31 (max 31) - age 1: 359488 bytes, 359488 total - age 2: 314104 bytes, 673592 total : 167034K-657K(169600K), 0.2717012 secs] 197223K-30846K(519168K), 0.2745516 secs] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start(Native Method) at org.apache.catalina.connector.http.HttpProcessor.threadStart(HttpProcessor.j ava:1178) at org.apache.catalina.connector.http.HttpProcessor.start(HttpProcessor.java:12 62) at org.apache.catalina.connector.http.HttpConnector.newProcessor(HttpConnector. java:937) at org.apache.catalina.connector.http.HttpConnector.createProcessor(HttpConnect or.java:868) at org.apache.catalina.connector.http.HttpConnector.run(HttpConnector.java:1075 ) at java.lang.Thread.run(Thread.java:534) thanks and regards, Javid -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 10:00 PM To: '[EMAIL PROTECTED]' Subject:RE: verbose gc we are using tomcat 4.1 withjdk1.3, we have folowing clarifications 1) [Full GC 34521K-15504K(38328K), 0.5953540 secs] [GC 27533K-16335K(38328K), 0.0396336 secs] is the statements got in catalina.out by adding -server -verbose:gc in catalina.sh 2) what does the above statements got in catalina.out means, is garbage collection active. When i execute top command on solaris and see the memory it shows size and resisdent memory , what
RE: verbose gc
Dhruva B. Reddy said: As I said in my post, we would get the OutOfMemoryError about every 24 hours. I tried setting MaxPermSize to 256MB (haven't been able to determine the default value for Sun's JVM, but it's 64MB for HP's JVM). This sets the size of the permanent generation, into which longer-lived objects and class definitions are loaded (we have over 3500 classes over multiple web apps). I believe the default maxperm size for sun jvm is 64MB. I ran into the same problem, when spawning additional threads, and too increased the maxperm to 256MB. that allows me to spawn up to 3,000 threads. I prefer to spawn all the threads at startup rather then have tomcat spawn threads as it needs them. my heap is set to 800MB for tomcat. the systems have between 2GB and 3GB of ram each. been running for ages without any problems. never had an out of memory message from tomcat(with the exceptions of when I was testing in a test enviornment with thread settings) running tomcat 4.0.6. Tried going to 4.1.27 but behavior was significantly worse than with 4.0.6 for the stuff I am using it with. running sun java 1.3_9 or something. have never run out of heap either. still looking for further tips/docs on how to optimize thread usage further. nate - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: verbose gc
Hi, Thanks for all the help. We did loadtesting of our application running on tomcat 4.1.27 with jdk1.4 on solaris, we enabled the GC stack to get details of how GC works, we ran for 1 users with 5 users logging every 30 secs. We set min and max heap size as 512mb The test ran perfectly for first 2 hours and total heap ocupied was only 10448K out of total of 519168K , later as test case progessed for around 10 hours the tomcat stopped responding and it has thrown OutOfMemoryError , but as can be seen from GC output before outofmemory error the total heap occupied was only 30846K, though we have lot of more space on heap still tomcat crashed and give connectiontimeout for remaining pages. When we see the size occupied from top command , when tomcat is started it was 622Mb and at time of crash it was 723MB. We are looking for following clarifications 1 What could be the reason for tomcat to crash though we have lot of heap remaining. 2 Why the memory occupied as seend from top command increased though there was lot of space on heap 3 We noticed that though heap has space when size seen from top command reached 720 to 270 mb tomcat crashes. 4 How to make tomcat running without crash , certianly heap is not problem in this case 5 Is this a known issue, Any optimization to be done to avoid this problem. The below is the output of the GC after which we get outofmemory error [GC 55404.101: [DefNew Desired survivor size 2621440 bytes, new threshold 31 (max 31) - age 1: 359488 bytes, 359488 total - age 2: 314104 bytes, 673592 total : 167034K-657K(169600K), 0.2717012 secs] 197223K-30846K(519168K), 0.2745516 secs] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start(Native Method) at org.apache.catalina.connector.http.HttpProcessor.threadStart(HttpProcessor.j ava:1178) at org.apache.catalina.connector.http.HttpProcessor.start(HttpProcessor.java:12 62) at org.apache.catalina.connector.http.HttpConnector.newProcessor(HttpConnector. java:937) at org.apache.catalina.connector.http.HttpConnector.createProcessor(HttpConnect or.java:868) at org.apache.catalina.connector.http.HttpConnector.run(HttpConnector.java:1075 ) at java.lang.Thread.run(Thread.java:534) thanks and regards, Javid -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 10:00 PM To: '[EMAIL PROTECTED]' Subject: RE: verbose gc we are using tomcat 4.1 withjdk1.3, we have folowing clarifications 1) [Full GC 34521K-15504K(38328K), 0.5953540 secs] [GC 27533K-16335K(38328K), 0.0396336 secs] is the statements got in catalina.out by adding -server -verbose:gc in catalina.sh 2) what does the above statements got in catalina.out means, is garbage collection active. When i execute top command on solaris and see the memory it shows size and resisdent memory , what is the diference between the two. 3) I see that gc statements are logged to catalina.out but the memory usage keeps increasing, does it mean there is problem with garbage collection, 4) can we get more information like how many obejcts created / objects destroyed during a gc run, what options to pass to get that information thanks -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 2:18 PM To: '[EMAIL PROTECTED]' Subject:verbose gc Hi, When we give verbose:gc in cataline.sh during startup of tomcat where does the gc information get logged. I am not seeing any information in catalina.out. thanks. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: verbose gc
Unable to create native thread - I had a quick crack at this one and you can up the amount of Threads available to Tomcat in server.xml. You can also pass in a -Xss parameter to the JVM re the thread stack size - these fixed my identical problem. No worries since. (Tomcat 5.0.1x though). Pete. Mohammed Javid - CTD, Chennai. wrote: Hi, Thanks for all the help. We did loadtesting of our application running on tomcat 4.1.27 with jdk1.4 on solaris, we enabled the GC stack to get details of how GC works, we ran for 1 users with 5 users logging every 30 secs. We set min and max heap size as 512mb The test ran perfectly for first 2 hours and total heap ocupied was only 10448K out of total of 519168K , later as test case progessed for around 10 hours the tomcat stopped responding and it has thrown OutOfMemoryError , but as can be seen from GC output before outofmemory error the total heap occupied was only 30846K, though we have lot of more space on heap still tomcat crashed and give connectiontimeout for remaining pages. When we see the size occupied from top command , when tomcat is started it was 622Mb and at time of crash it was 723MB. We are looking for following clarifications 1 What could be the reason for tomcat to crash though we have lot of heap remaining. 2 Why the memory occupied as seend from top command increased though there was lot of space on heap 3 We noticed that though heap has space when size seen from top command reached 720 to 270 mb tomcat crashes. 4 How to make tomcat running without crash , certianly heap is not problem in this case 5 Is this a known issue, Any optimization to be done to avoid this problem. The below is the output of the GC after which we get outofmemory error [GC 55404.101: [DefNew Desired survivor size 2621440 bytes, new threshold 31 (max 31) - age 1: 359488 bytes, 359488 total - age 2: 314104 bytes, 673592 total : 167034K-657K(169600K), 0.2717012 secs] 197223K-30846K(519168K), 0.2745516 secs] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start(Native Method) at org.apache.catalina.connector.http.HttpProcessor.threadStart(HttpProcessor.j ava:1178) at org.apache.catalina.connector.http.HttpProcessor.start(HttpProcessor.java:12 62) at org.apache.catalina.connector.http.HttpConnector.newProcessor(HttpConnector. java:937) at org.apache.catalina.connector.http.HttpConnector.createProcessor(HttpConnect or.java:868) at org.apache.catalina.connector.http.HttpConnector.run(HttpConnector.java:1075 ) at java.lang.Thread.run(Thread.java:534) thanks and regards, Javid -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 10:00 PM To: '[EMAIL PROTECTED]' Subject: RE: verbose gc we are using tomcat 4.1 withjdk1.3, we have folowing clarifications 1) [Full GC 34521K-15504K(38328K), 0.5953540 secs] [GC 27533K-16335K(38328K), 0.0396336 secs] is the statements got in catalina.out by adding -server -verbose:gc in catalina.sh 2) what does the above statements got in catalina.out means, is garbage collection active. When i execute top command on solaris and see the memory it shows size and resisdent memory , what is the diference between the two. 3) I see that gc statements are logged to catalina.out but the memory usage keeps increasing, does it mean there is problem with garbage collection, 4) can we get more information like how many obejcts created / objects destroyed during a gc run, what options to pass to get that information thanks -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 2:18 PM To: '[EMAIL PROTECTED]' Subject: verbose gc Hi, When we give verbose:gc in cataline.sh during startup of tomcat where does the gc information get logged. I am not seeing any information in catalina.out. thanks. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] *** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. *** For any information on the Quinn Group of Companies please visit :- http://www.quinn-group.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: verbose gc
The heap size has nothing to do with the memory size that is seen by the system. You have to look at least at the total memory. (That is used + free memory) To that you have to add - thread stacks (At least some vm's don't allocate them on the heap) - static memory (Like the jvm itself, static strings, classes, jars, ...) - some os memory that is used by the vm to manage it self - ... Size vs. resident size is the total size vs. the used size as seen by the os. How many threads where active at the time of the error ? The vm may have hit some os limits on the number of active threads (either system wide or per owner) The vm may have run out of non heap memory. (e.g.: Threadstacksize) -Original Message- From: Mohammed Javid - CTD, Chennai. [mailto:[EMAIL PROTECTED] Sent: Thursday, January 29, 2004 10:35 AM To: [EMAIL PROTECTED] Subject: RE: verbose gc but as can be seen from GC output before outofmemory error the total heap occupied was only 30846K, though we have lot of more space on heap still tomcat crashed and give connectiontimeout for remaining pages. When we see the size occupied from top command , when tomcat is started it was 622Mb and at time of crash it was 723MB. 1) [Full GC 34521K-15504K(38328K), 0.5953540 secs] [GC 27533K-16335K(38328K), 0.0396336 secs] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: verbose gc
Mohammed, 1) [Full GC 34521K-15504K(38328K), 0.5953540 secs] [GC 27533K-16335K(38328K), 0.0396336 secs] 2) what does the above statements got in catalina.out means, is garbage collection active? Yes, garbale collection is active. When the garbage collector runs, it will print a line out to stdout (or stderr?) which ends up in your catalina.out. If it runs a 'full' GC (where it takes a lot of time and cleans up a lot of stuff), then it'll say Full GC, like the first example. Otherwise, it's doing an incremental GC and not hitting everything, like the second example. In a healthy syste, you should see lots of regular GCs, and occationally see a full GC. The two big numbers indicate the size of the active heap (the total size of 'live' objects before and after the GC). So, in the first one, the heap was reduced from 34521k of active objects to 15505k of active objects. The number in parens () is the size of the non-permanent heap, which is the total heap minus one of the surviver spaces. The last number is the wall-clock time that the GC was active -- it tools half a second to so a full GC and about 1/20 of a second to do the incremental GC. Please see http://java.sun.com/docs/hotspot/gc1.4.2/#2.2.%20Measurement|outline for more information. When i execute top command on solaris and see the memory it shows size and resisdent memory , what is the diference between the two. I think this might mean different things to different OSs, but generally the size is the total size of executable + code + heap for that process -- everything it needs to run. The 'resident' is usually the amount of 'size' that's actually in physical RAM and not swapped out to disk by the OS. 3) I see that gc statements are logged to catalina.out but the memory usage keeps increasing, does it mean there is problem with garbage collection, When you say 'memory usage', do you mean from top? Right. Java doesn't return memory to the OS once it requests it -- there's generally no need to do that. Even though Java's heap has more 'space' left over in it, it only has space left over for more Java objects. You should not be comparing the Java heap and GCs to what you read out of 'top'. You should use Runtime.freeMemory (the available heap size), Runtime.totalMemory (the total heap size + internal VM memory), and Runtime.maxMemory (the maximum amount of memory that the VM will try to use). 4) can we get more information like how many obejcts created / objects destroyed during a gc run, what options to pass to get that information You need a profiler to do heap inspections, which has been discussed many times on this list. I'm not sure you can find out exactly what the GC is doing... you might need a very nice profiler for that. -chris - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: verbose gc
I have an application stdout (in linux), and it goes there not catalina.out. If you are running Tomcat as a service, I think that could be different. Pete. Mohammed Javid - CTD,To: [EMAIL PROTECTED] Chennai. cc: [EMAIL PROTECTED] Subject: verbose gc ltech.com 14/01/2004 08:48 Please respond to Tomcat Users List Hi, When we give verbose:gc in cataline.sh during startup of tomcat where does the gc information get logged. I am not seeing any information in catalina.out. thanks. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] *** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. *** For any information on the Quinn Group of Companies please visit :- http://www.quinn-group.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: verbose gc
we are using tomcat 4.1 withjdk1.3, we have folowing clarifications 1) [Full GC 34521K-15504K(38328K), 0.5953540 secs] [GC 27533K-16335K(38328K), 0.0396336 secs] is the statements got in catalina.out by adding -server -verbose:gc in catalina.sh 2) what does the above statements got in catalina.out means, is garbage collection active. When i execute top command on solaris and see the memory it shows size and resisdent memory , what is the diference between the two. 3) I see that gc statements are logged to catalina.out but the memory usage keeps increasing, does it mean there is problem with garbage collection, 4) can we get more information like how many obejcts created / objects destroyed during a gc run, what options to pass to get that information thanks -Original Message- From: Mohammed Javid - CTD, Chennai. Sent: Wednesday, January 14, 2004 2:18 PM To: '[EMAIL PROTECTED]' Subject: verbose gc Hi, When we give verbose:gc in cataline.sh during startup of tomcat where does the gc information get logged. I am not seeing any information in catalina.out. thanks. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
