RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544)
of the velocity page so that the Velocity RunTime doesn't have to parse these backslashes. Hopefully this will cure the problem. I just thought I will post this just in case some of you are seeing similar problems in your app. Good luck. Arup -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: 20 July 2005 17:45 To: Tomcat Users List; [EMAIL PROTECTED] Subject: RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) From: Arup Vidyerthy [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) Any pointers in terms of how I should approach this problem...? I'd raise the limit on the number of open file descriptors and see what happens. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ___ To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544)
of the velocity page so that the Velocity RunTime doesn't have to parse these backslashes. Hopefully this will cure the problem. I just thought I will post this just in case some of you are seeing similar problems in your app. Good luck. Arup -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: 20 July 2005 17:45 To: Tomcat Users List; [EMAIL PROTECTED] Subject: RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) From: Arup Vidyerthy [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) Any pointers in terms of how I should approach this problem...? I'd raise the limit on the number of open file descriptors and see what happens. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ___ To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ___ Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
Hi Steve, We're probably going to need a little more context in order to give you any help. Approximately how often does this happen? Have you been able to isolate a particular page or servlet that causes it consistently? What shows up in the log file/console window when it happens? Does it kill just the request, or the app, or the entire server? I'll give you a couple possibilities to consider (I'm sure smarter people than I will provide others): - A JSP or Servlet with an infinite loop may exhaust the memory and cause the issue, either on that page or (possibly) on other pages while that one is in the process of dying. - You may have a memory leak in one of your apps of a driver (yes, they're possible in java); you can check for symptoms of this by watching the memory usage after a clean start and seeing if it continually goes up by small amounts over time. Even though you have 600 free megs of physical memory I don't believe the JVM will consume all of that by default. Anyway, let us know details. ~PST On 8/5/05, Steve Sheerin [EMAIL PROTECTED] wrote: From: Steve Sheerin Sent: Thursday, August 04, 2005 2:29 PM To: '[EMAIL PROTECTED]' Subject: OutOfMemoryError I have a new Tomcat based server that is coming up with this error when communicating with the server. Most of the time it works great, but every now and then we get this message. In checking the server, it's memory looks fine - base is 1 gig, with 600 meg avail. Any ideas Steve Sheerin Clark Public Utilities 360-992-3243 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544)
From: Arup Vidyerthy [mailto:[EMAIL PROTECTED] Subject: FW: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) Problem is every now and then we will see an OutOfMemoryError on the box that looks similar to this: java.lang.OutOfMemoryError java.util.zip.ZipFile.open(Native Method) Unfortunately, OOME is used as a catch-all for many OS-related resource problems. My guess is in this case that you've exceeded the limit on file descriptors for the process. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544)
Hi Chuck, Thanks for your reply. In other words, may be not have 15 contexts/web app on one server. May be 10. I guess it is hard to say. Looking at the no. of jars, do you think it is an excessive amount? Any pointers in terms of how I should approach this problem...? Thanks... /Arup -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: 20 July 2005 17:14 To: Tomcat Users List Subject: RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) From: Arup Vidyerthy [mailto:[EMAIL PROTECTED] Subject: FW: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) Problem is every now and then we will see an OutOfMemoryError on the box that looks similar to this: java.lang.OutOfMemoryError java.util.zip.ZipFile.open(Native Method) Unfortunately, OOME is used as a catch-all for many OS-related resource problems. My guess is in this case that you've exceeded the limit on file descriptors for the process. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ___ Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544)
From: Arup Vidyerthy [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError WebappClassLoader.openJARs(WebappClassLoader.java:1544) Any pointers in terms of how I should approach this problem...? I'd raise the limit on the number of open file descriptors and see what happens. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
We're battling this same problem at our client's site. I think I've identified the same cause (running out of Perm Gen space) but I have to prove it to them first. Hooking a profiler into the Tomcat instance should clearly show this but they don't have one handy. Moving some of the applications to a second instance of Tomcat should also solve this problem. -- Virtually, Andre Van Klaveren Architect III, SCP Enterprise Transformation Services Unisys Corporation On 4/21/05, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? the default Perm is 64M. If LeeAnn's webapp have a large number of classes, maybe there's too many; therefore forcing the VM to resize the perm. IIRC (haven't looked at core HotSpot code in detail since 1.4.1 days), the virtual address boundary between perm and the other generations is fixed at JVM initialization and never moves. The perm gen does go through GC just like the other gens, but no memory ever moves between the perm and any other space, and the perm gen maximum virtual address range never changes. The HotSpot JVM allocates, but does not commit, all of the -Xmx space at startup in one contiguous block (although there were rumors of that changing in later JVMs). Only the -Xms portion is committed, split up into three regions - young, old, and perm. When a region fills and GC doesn't reduce the usage significantly, additional pages are committed in whichever region needs expanding. I think what was happening is that LeeAnn simply had more classes than would fit in 64 MB. Nothing to do with threads, just the total number of classes in all the applications added together. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
the advice given in previous message to dump the Perm gen info out should tell you. running tomcat in a profiler may not show it. I use optimizeIt frequently and it doesn't show perm generation. peter On 4/27/05, Andre Van Klaveren [EMAIL PROTECTED] wrote: We're battling this same problem at our client's site. I think I've identified the same cause (running out of Perm Gen space) but I have to prove it to them first. Hooking a profiler into the Tomcat instance should clearly show this but they don't have one handy. Moving some of the applications to a second instance of Tomcat should also solve this problem. -- Virtually, Andre Van Klaveren Architect III, SCP Enterprise Transformation Services Unisys Corporation On 4/21/05, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? the default Perm is 64M. If LeeAnn's webapp have a large number of classes, maybe there's too many; therefore forcing the VM to resize the perm. IIRC (haven't looked at core HotSpot code in detail since 1.4.1 days), the virtual address boundary between perm and the other generations is fixed at JVM initialization and never moves. The perm gen does go through GC just like the other gens, but no memory ever moves between the perm and any other space, and the perm gen maximum virtual address range never changes. The HotSpot JVM allocates, but does not commit, all of the -Xmx space at startup in one contiguous block (although there were rumors of that changing in later JVMs). Only the -Xms portion is committed, split up into three regions - young, old, and perm. When a region fills and GC doesn't reduce the usage significantly, additional pages are committed in whichever region needs expanding. I think what was happening is that LeeAnn simply had more classes than would fit in 64 MB. Nothing to do with threads, just the total number of classes in all the applications added together. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
So let me get this straight, LeeAnn is specifying a maximum heap size of 512MB and a minimum size of 128MB. It looks like the heap doesn't get adjusted up when the servlets are initializing? Thus she needs to increase her minimum heapsize, so something like -Xms512m should fix it? -ryan -Original Message- From: Peter Lin [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 20, 2005 8:35 PM To: Tomcat Users List Subject: Re: OutOfMemoryError - 100 thread limit? ok, I just tried starting tomcat with the default heap size and 18 instances of my webapp and it got OOME just as you see. If I set my heap to -Xms256m -Xmx512m I'm able to load the 18 webapps just fine. the total memory used after all the webapps are loaded is 152Megs. the total number of webapps is 25 webapps including the default tomcat webapps and a few more. so basically you'll need to set the heap larger enough and it should work. If i do some math, we can calculate a rough estimate for the min heap (154 - 64)/25 = 3.6megs so basically, you'll need to multiply the number of webapps by 4 megs to figure out the heap setting. peter lin On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: when you say 50 threads, do you mean 50 separate web applications? My concern right now is that we seem to be limited to 17-18 copies of our web app on a particular installation of tomcat. And that is just hitting the login page - not doing any work. Practically, for applications with many users, we may be limited to 7-10 copies of our application on a tomcat. Maybe I'm expecting too much? Perhaps that's a reasonable number of concurrent webapps that can be run on a tomcat? But if that is the case, it would be good to know what the limiting factor is, if it's not number of threads or amount of memory. thanks for much for your help! At 05:48 PM 4/20/2005, you wrote: ahh ok. .. my confusion. back to the problem you see. I've tested tomcat with 30-40 threads without any problems in the past. Even with heavy weight JSTL tags in JSP's, I'm able to go up to 50 threads with tomcat4.1. I'll try hitting tomcat's status servlet with 50 threads tonight and see what happens. peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: I think we've gotten things a bit confused :) The application itself is not thread intensive - our original tests show that each separate copy of the web application started up 3 active threads. Further testing, when we added in debugging code that would spin up dozens of threads just to see if we could cause the tomcat to get the error if we started one application up and had tons of threads, showed that we could get 800+ threads with no problems (no outofmemory errors) and that the problem does not in fact seem to be specifically related to the number of threads the application creates. In fact, regardless of whether each instance of the application spins up 3 threads (real situation) or 10+ (debug code only) threads, we still get the out of memory error not when we hit a certain number of threads, but actually when we hit a certain number of servlets having been initialized. Testing a very simple servlet, I can start up 25 instances of the servlet with no OOM error. Testing our application's servlet, I can only start up 17-18 instances of the servlet before getting an OOM error. At this point I am going through our standard application servlet, trying to isolate what work it does that may cause a problem when we do it 17 or 18 times on one installation of tomcat. At 05:16 PM 4/20/2005, you wrote: if your application is thread heavy, I would recommend changing it so that one thread can manage several processes. creating 800+ threads is going to hit a scalability and reliability wall very quickly. In general, you want to make sure the number of threads remain constant under constant load. If it doesn't the server will eventually crash as you currently see. I'm guessing the application is using threads to handle async processes, which is good, but spawning new threads with every request isn't going to work. hope that helps peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs
RE: OutOfMemoryError - 100 thread limit?
From: J. Ryan Earl [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError - 100 thread limit? So let me get this straight, LeeAnn is specifying a maximum heap size of 512MB and a minimum size of 128MB. It looks like the heap doesn't get adjusted up when the servlets are initializing? Thus she needs to increase her minimum heapsize, so something like -Xms512m should fix it? Probably not, since I suspect she's running out of perm gen space. IIRC, the perm gen size is calculated from -Xmx, which is already at 512m. Using the PrintGCDetails will tell us for sure. If that shows perm gen space to be exhausted, then adjusting -XX:MaxPermSize should fix it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
my 2 bits. when I tried to replicate the issue LeeAnn saw, it was pretty clearn the JVM can't resize the heap fast enough to account for the large number of webapps being loaded. simply increasing the initial heap should solve the problem. peter On 4/21/05, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: J. Ryan Earl [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError - 100 thread limit? So let me get this straight, LeeAnn is specifying a maximum heap size of 512MB and a minimum size of 128MB. It looks like the heap doesn't get adjusted up when the servlets are initializing? Thus she needs to increase her minimum heapsize, so something like -Xms512m should fix it? Probably not, since I suspect she's running out of perm gen space. IIRC, the perm gen size is calculated from -Xmx, which is already at 512m. Using the PrintGCDetails will tell us for sure. If that shows perm gen space to be exhausted, then adjusting -XX:MaxPermSize should fix it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
Peter Lin reproduced and fixed the problem LeeAnn is seeing, and said If I set my heap to -Xms256m -Xmx512m I'm able to load the 18 webapps just fine. the total memory used after all the webapps are loaded is 152Megs. He didn't mention anything about adjusting the perm gen space. -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 9:19 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: J. Ryan Earl [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError - 100 thread limit? So let me get this straight, LeeAnn is specifying a maximum heap size of 512MB and a minimum size of 128MB. It looks like the heap doesn't get adjusted up when the servlets are initializing? Thus she needs to increase her minimum heapsize, so something like -Xms512m should fix it? Probably not, since I suspect she's running out of perm gen space. IIRC, the perm gen size is calculated from -Xmx, which is already at 512m. Using the PrintGCDetails will tell us for sure. If that shows perm gen space to be exhausted, then adjusting -XX:MaxPermSize should fix it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
for what it's worth, adjusting the perm gen space rarely helps. I did a ton of benchmarks back in 2002 using all sorts of -X combinations and it rarely helped. You have to really know the memory allocation pattern to effectively tune perm gen space. adjusting the new eden may help, but again that requires quite a bit of profiling to really understand how an application is allocating memory. peter On 4/21/05, J. Ryan Earl [EMAIL PROTECTED] wrote: Peter Lin reproduced and fixed the problem LeeAnn is seeing, and said If I set my heap to -Xms256m -Xmx512m I'm able to load the 18 webapps just fine. the total memory used after all the webapps are loaded is 152Megs. He didn't mention anything about adjusting the perm gen space. -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 9:19 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: J. Ryan Earl [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError - 100 thread limit? So let me get this straight, LeeAnn is specifying a maximum heap size of 512MB and a minimum size of 128MB. It looks like the heap doesn't get adjusted up when the servlets are initializing? Thus she needs to increase her minimum heapsize, so something like -Xms512m should fix it? Probably not, since I suspect she's running out of perm gen space. IIRC, the perm gen size is calculated from -Xmx, which is already at 512m. Using the PrintGCDetails will tell us for sure. If that shows perm gen space to be exhausted, then adjusting -XX:MaxPermSize should fix it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: J. Ryan Earl [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError - 100 thread limit? Peter Lin reproduced and fixed the problem LeeAnn is seeing, and said If I set my heap to -Xms256m -Xmx512m I'm able to load the 18 webapps just fine. the total memory used after all the webapps are loaded is 152Megs. He didn't mention anything about adjusting the perm gen space. Nor is he running the same webapps that LeeAnn is using (as far as I know), so that's a bit of an apples and oranges comparison. Also, his conjecture that the JVM couldn't keep up doesn't fit with my understanding of object allocation and GC in a HotSpot JVM. If needed, the JVM simply suspends all the application threads and does a full GC. Again, the -XX:+PrintGCDetails will tell us what's really going on. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
I may not be running the same app LeeAnn is running, but my webapp is fairly heavy and creates several threads per webapp. Even if it's not an apples-to-apples comparison, it's still a worth while trick to try. if it doesn't work, then back to square one :) if it does, then LeeAnn can go back to work. peter On 4/21/05, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: J. Ryan Earl [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError - 100 thread limit? Peter Lin reproduced and fixed the problem LeeAnn is seeing, and said If I set my heap to -Xms256m -Xmx512m I'm able to load the 18 webapps just fine. the total memory used after all the webapps are loaded is 152Megs. He didn't mention anything about adjusting the perm gen space. Nor is he running the same webapps that LeeAnn is using (as far as I know), so that's a bit of an apples and oranges comparison. Also, his conjecture that the JVM couldn't keep up doesn't fit with my understanding of object allocation and GC in a HotSpot JVM. If needed, the JVM simply suspends all the application threads and does a full GC. Again, the -XX:+PrintGCDetails will tell us what's really going on. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? if it doesn't work, then back to square one :) It would be even better to get some real data with the -XX:+PrintGCDetails, so we can stop speculating... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
I, for one, am awaiting LeeAnn's response with abated breath. =) -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 9:53 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? if it doesn't work, then back to square one :) It would be even better to get some real data with the -XX:+PrintGCDetails, so we can stop speculating... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
Good Morning everyone :) Apparently I'm one of the few on the west coast - sorry for the delay this morning. I added the PrintGCDetails line to my CATALINA_OPTS and got a bit more information - although I'm afraid I'm not sure if this is the information Chuck was looking for? My catalina.out file showed the following lines just before tomcat tipped over at 17 sites: Attempting to Open Log File: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site17/WEB-INF/logs/EVJ.log Successfully finished init in SesameServlet. [Full GC [Tenured: 43103K-45066K(116544K), 0.6647620 secs] 55017K-45066K(129792K), 0.6648170 secs] [GC [DefNew: 11826K-1236K(13248K), 0.0184770 secs] 56893K-46303K(129792K), 0.0185260 secs] Start Init Method. The CONFIG_FILE from web.xml is : WEB-INF/configuration/Configuration.properties Using Configuration FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF/configuration/Configuration.properties Successfully created a SesameConfig object from FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF/configuration/Configuration.properties SesameProjectDir: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF Attempting to Open Log File: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF/logs/EVJ.log Successfully finished init in SesameServlet. [GC [DefNew: 13076K-686K(13248K), 0.0199470 secs] 58143K-46974K(129792K), 0.0199950 secs] [GC [DefNew: 12523K-1408K(13248K), 0.0300140 secs] 58810K-48267K(129792K), 0.0300740 secs] [Full GC [Tenured: 46859K-48385K(116544K), 0.6326870 secs] 50745K-48385K(129792K), 0.6327540 secs] [Full GC [Tenured: 48385K-47888K(116544K), 0.7671290 secs] 48385K-47888K(129792K), 0.7671810 secs] [Full GC [Tenured: 47888K-47924K(116544K), 0.6296020 secs] 48999K-47924K(129792K), 0.6296660 secs] [Full GC [Tenured: 47924K-47930K(116544K), 0.6508840 secs] 47939K-47930K(129792K), 0.6509620 secs] [Full GC [Tenured: 47930K-47930K(116544K), 0.6475860 secs] 47931K-47930K(129792K), 0.6476490 secs] [Full GC [Tenured: 47930K-47912K(116544K), 0.7901650 secs] 47932K-47912K(129792K), 0.7902320 secs] I will try increasing the Xms parameter now, although I believe we did try that earlier in the debugging process. More in a bit. At 08:25 AM 4/21/2005, you wrote: I, for one, am awaiting LeeAnn's response with abated breath. =) -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 9:53 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? if it doesn't work, then back to square one :) It would be even better to get some real data with the -XX:+PrintGCDetails, so we can stop speculating... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
Ok, this time I used -Xms512m -Xmx512m with the same results - Successfully created a SesameConfig object from FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF/configuration/Configuration.properties SesameProjectDir: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF [Full GC [Tenured: 45429K-48110K(466048K), 0.6853640 secs] 80819K-48110K(518464K), 0.6854250 secs] [Full GC [Tenured: 48110K-47709K(466048K), 0.7746580 secs] 48119K-47709K(518464K), 0.7747100 secs] Right on the 18th site spin-up - I'm not getting OutOfMemory errors in the catalina.out trace, but the servlet didn't load and gave me an error in the browser: pe Exception report message description The server encountered an internal error () that prevented it from fulfilling this request. exception javax.servlet.ServletException: Servlet.init() for servlet ExtraView threw exception org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:300) org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:374) org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743) org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675) org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) java.lang.Thread.run(Thread.java:536) root cause java.lang.OutOfMemoryError note The full stack trace of the root cause is available in the Apache Tomcat/5.0.28 logs. At 08:25 AM 4/21/2005, you wrote: I, for one, am awaiting LeeAnn's response with abated breath. =) -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 9:53 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? if it doesn't work, then back to square one :) It would be even better to get some real data with the -XX:+PrintGCDetails, so we can stop speculating... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115
Re: OutOfMemoryError - 100 thread limit?
wow, your webapp must be pretty heavy and have lots of jar and/or classes. The behavior that I saw was that 16 of the webapps would load with the default JVM heap settings. When I went to Tomcat's manager page, and click start, I got a similar OOME and the webapp would not start. At that point, there was nothing I could do to get it to start. once I shut down tomcat, change the heap and restart, tomcat was able to load all the webapps. Here's a trick that might help. Start tomcat, but with only 10 copies of your webapp. look at the total memory used and substract 64meg. Divide that number by 10 and see what the memory requirements are for each webapp. (Total mem - 64mb ) / number of webapps = mem per webapp. once you have that, you can figure what you really need. if the total for 18 webapps is less than 512Mb, it means something else is happening, which means you can rule out Tomcat as the cause. Since all tomcat does is start the webapp and call the servletContextListeners, it really isn't doing much. hope that helps peter On 4/21/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: Ok, this time I used -Xms512m -Xmx512m with the same results - Successfully created a SesameConfig object from FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF/configuration/Configuration.properties SesameProjectDir: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF [Full GC [Tenured: 45429K-48110K(466048K), 0.6853640 secs] 80819K-48110K(518464K), 0.6854250 secs] [Full GC [Tenured: 48110K-47709K(466048K), 0.7746580 secs] 48119K-47709K(518464K), 0.7747100 secs] Right on the 18th site spin-up - I'm not getting OutOfMemory errors in the catalina.out trace, but the servlet didn't load and gave me an error in the browser: pe Exception report message description The server encountered an internal error () that prevented it from fulfilling this request. exception javax.servlet.ServletException: Servlet.init() for servlet ExtraView threw exception org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:300) org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:374) org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743) org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675) org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) java.lang.Thread.run(Thread.java:536) root cause java.lang.OutOfMemoryError note The full stack trace of the root cause is available in the Apache Tomcat/5.0.28 logs. At 08:25 AM 4/21/2005, you wrote: I, for one, am awaiting LeeAnn's response with abated breath. =) -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 9:53 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? if it doesn't work, then back to square one :) It would be even better to get some real data with the -XX:+PrintGCDetails, so we can stop speculating... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
silly question, does this webapp have like thousands of JSP and servlets and preload the JSP's? peter - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
Not a silly question :) We have 1 servlet that is the main entry point, everything else is done with standard java classes. We have no jsp's. At 09:13 AM 4/21/2005, you wrote: silly question, does this webapp have like thousands of JSP and servlets and preload the JSP's? peter - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
Possible success! I went back to my weblogic installation which I was successfully able to open up 25 sites to see what it was doing when starting up. It had -XX:MaxPermSize=128m in the process description. I added that to my catalina_opts and I got past the 18 site limitation of just firing them up. Can anyone give me any information on what this does? Is this the perm gen that Chuck mentioned before? Are there any caveats to setting this memory option? thanks! LeeAnn At 09:13 AM 4/21/2005, you wrote: silly question, does this webapp have like thousands of JSP and servlets and preload the JSP's? peter - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
glad I was wrong and you found a solution. http://java.sun.com/docs/hotspot/VMOptions.html I'm just guessing here, but by forcing the maxPermSize to 128, it leaves more space for the eden and prevents classes from getting promoted to perm. Atleast that's logical explanation I can think of. Another way to get the heap to resize in larger increments and prevent it from getting smaller is to set the -XX:MinHeapFreeRatio. peter lin On 4/21/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: Possible success! I went back to my weblogic installation which I was successfully able to open up 25 sites to see what it was doing when starting up. It had -XX:MaxPermSize=128m in the process description. I added that to my catalina_opts and I got past the 18 site limitation of just firing them up. Can anyone give me any information on what this does? Is this the perm gen that Chuck mentioned before? Are there any caveats to setting this memory option? thanks! LeeAnn At 09:13 AM 4/21/2005, you wrote: silly question, does this webapp have like thousands of JSP and servlets and preload the JSP's? peter - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
This is a really interesting and confusing problem... Can you tell how much memory the OS is actually committing to Java? ie Check the resident stable set (RSS) = how much of the virtual memory space has been physically commited to memory. # ps axl|grep java 0 501 2623 1 21 0 1264252 79112 - Sl ? 0:09 /usr/java/jre1.5.0_02/bin/java -Xmx1000m The above says I have 1264252KB allocated but only 79112KB committed to physical memory. I'm curious what yours looks like. I know the the JVM reported it was using less memory, but I'd be curious to see what the OS thinks. Maybe even try raising your maximum heap size to say 1024MB? Which JVM are you using again? Which other JVMs have you tried? -ryan PS I just checked a production instance, and it looks like: $ ps axl|grep java 0 101 24243 1 16 0 2275516 906312 322497 Sl ?465:48 /opt/sun-jdk-1.4.2.07/bin/java ~900MB of over 2GB of virtual memory commited, it's apparently got quite a few threads running: $ ps ax -L |grep java|wc -l 2016 How does your application compare when it runs out of memory? -Original Message- From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 11:21 AM To: Tomcat Users List Subject: Re: OutOfMemoryError - 100 thread limit? Not a silly question :) We have 1 servlet that is the main entry point, everything else is done with standard java classes. We have no jsp's. At 09:13 AM 4/21/2005, you wrote: silly question, does this webapp have like thousands of JSP and servlets and preload the JSP's? peter - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? It had -XX:MaxPermSize=128m in the process description. Ahah, as they say. Can anyone give me any information on what this does? Is this the perm gen that Chuck mentioned before? Yes, this controls the size of the permanent generation. The permanent generation portion of the heap is where class objects (instances of java.lang.Class) are allocated. If you have a lot of classes - which app servers frequently do - the size of this space needs to be increased. A quick scan of the 1.4.2 HotSpot source shows the default value to be 64m. Are there any caveats to setting this memory option? Other than it subtracts from the total heap space for normal object, no. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
Awesome, I'm glad that was solved as I'm about to do the same thing with our application: running multiple instances of the application on a single tomcat instance housing many virtual hosts. Thanks for the http://java.sun.com/docs/hotspot/VMOptions.html link Peter. -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 11:52 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? It had -XX:MaxPermSize=128m in the process description. Ahah, as they say. Can anyone give me any information on what this does? Is this the perm gen that Chuck mentioned before? Yes, this controls the size of the permanent generation. The permanent generation portion of the heap is where class objects (instances of java.lang.Class) are allocated. If you have a lot of classes - which app servers frequently do - the size of this space needs to be increased. A quick scan of the 1.4.2 HotSpot source shows the default value to be 64m. Are there any caveats to setting this memory option? Other than it subtracts from the total heap space for normal object, no. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? I'm just guessing here, but by forcing the maxPermSize to 128, it leaves more space for the eden and prevents classes from getting promoted to perm. I think you're confusing tenured with perm. Nothing is ever promoted to perm - only known long-lived objects (primarily instances of java.lang.Class) are allocated in there, and no objects ever migrate to or from perm space. Regular objects are normally allocated in eden space (young gen), and if they live long enough, are migrated to the tenured area (old gen) during a GC. Increasing the perm size actually reduces, not increases, the amount of eden space available since the total of all three generations (young, old, perm) must come from the -Xmx setting. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
glad I can help. it's the little odd edge case problems that I find most interesting. it was fun to figure why that behavior was occuring. peter On 4/21/05, J. Ryan Earl [EMAIL PROTECTED] wrote: Awesome, I'm glad that was solved as I'm about to do the same thing with our application: running multiple instances of the application on a single tomcat instance housing many virtual hosts. Thanks for the http://java.sun.com/docs/hotspot/VMOptions.html link Peter. -ryan -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Thursday, April 21, 2005 11:52 AM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? It had -XX:MaxPermSize=128m in the process description. Ahah, as they say. Can anyone give me any information on what this does? Is this the perm gen that Chuck mentioned before? Yes, this controls the size of the permanent generation. The permanent generation portion of the heap is where class objects (instances of java.lang.Class) are allocated. If you have a lot of classes - which app servers frequently do - the size of this space needs to be increased. A quick scan of the 1.4.2 HotSpot source shows the default value to be 64m. Are there any caveats to setting this memory option? Other than it subtracts from the total heap space for normal object, no. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
yup you're right, silly me. ignore my brain dead remark. as long as the problem gets solved, I don't mind looking stupid. so it would appear by setting the PermSize, the jvm is pushing all java.lang.Class instances to Perm rather than loading them in eden and then promoting them to tenured. If you run verbosegc again, the gc interval should be longer than previous settings. I can't think of any real trade-offs of setting Perm to 128 other than using more RAM. If I remember correctly Perm is also where static stuff goes. peter On 4/21/05, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? I'm just guessing here, but by forcing the maxPermSize to 128, it leaves more space for the eden and prevents classes from getting promoted to perm. I think you're confusing tenured with perm. Nothing is ever promoted to perm - only known long-lived objects (primarily instances of java.lang.Class) are allocated in there, and no objects ever migrate to or from perm space. Regular objects are normally allocated in eden space (young gen), and if they live long enough, are migrated to the tenured area (old gen) during a GC. Increasing the perm size actually reduces, not increases, the amount of eden space available since the total of all three generations (young, old, perm) must come from the -Xmx setting. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
Thanks everyone! I really appreciate all your ideas and input and experience! I still am not completely clear on eden, gen, etc - obviously I need to go research how memory allocation is done in java. But, after setting the maxPermSize, I was able to log in and do work on 18 sites which is more than I could do before :) We will be scheduling some load testing next week to see if 128 is the right number for our real life problem, but at least I can see the light at the end of the tunnel now! thank you all again for all your help! At 10:35 AM 4/21/2005, you wrote: yup you're right, silly me. ignore my brain dead remark. as long as the problem gets solved, I don't mind looking stupid. so it would appear by setting the PermSize, the jvm is pushing all java.lang.Class instances to Perm rather than loading them in eden and then promoting them to tenured. If you run verbosegc again, the gc interval should be longer than previous settings. I can't think of any real trade-offs of setting Perm to 128 other than using more RAM. If I remember correctly Perm is also where static stuff goes. peter On 4/21/05, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? I'm just guessing here, but by forcing the maxPermSize to 128, it leaves more space for the eden and prevents classes from getting promoted to perm. I think you're confusing tenured with perm. Nothing is ever promoted to perm - only known long-lived objects (primarily instances of java.lang.Class) are allocated in there, and no objects ever migrate to or from perm space. Regular objects are normally allocated in eden space (young gen), and if they live long enough, are migrated to the tenured area (old gen) during a GC. Increasing the perm size actually reduces, not increases, the amount of eden space available since the total of all three generations (young, old, perm) must come from the -Xmx setting. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? so it would appear by setting the PermSize, the jvm is pushing all java.lang.Class instances to Perm rather than loading them in eden and then promoting them to tenured. Instances of java.lang.Class are always allocated in the perm gen, never in an eden space, regardless of any heap or gen size settings. I can't think of any real trade-offs of setting Perm to 128 other than using more RAM. Unless you increase -Xmx correspondingly, enlarging perm space will reduce the amount available for regular objects. Generally, if you have the memory, increase -Xmx to whatever you can before swapping starts. If I remember correctly Perm is also where static stuff goes. I believe that's true, since static data is associated with java.lang.Class instances. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
On 4/21/05, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? so it would appear by setting the PermSize, the jvm is pushing all java.lang.Class instances to Perm rather than loading them in eden and then promoting them to tenured. Instances of java.lang.Class are always allocated in the perm gen, never in an eden space, regardless of any heap or gen size settings. logically, it makes sense that all java.lang.Class classes are loaded in Perm. But the result on the behavior LeeAnn saw is a bit perplexing. I'm just exploring here, so if it doesn't make sense, it's cuz I'm making wild guesses. ie, talking out of my rear. the default Perm is 64M. If LeeAnn's webapp have a large number of classes, maybe there's too many; therefore forcing the VM to resize the perm. Doing that, the VM probably is allocating a big block and copying all the data over. I believe that is much heavier task than an incremental GC and probably equally costly as a full GC. So once the new Perm is ready, the VM should make the old block available. So the end result of setting -XX:MaxPermSize is it avoid resizing Perm, which allows Tomcat to load the webapps quickly. Even though setting max Perm to 128 reduces the overall space for new/old generation, it results in better performance because the VM doesn't have to move perm? of course this is assuming the VM prefers to have a continuous block for Perm memory. I believe that's true, since static data is associated with java.lang.Class instances. - Chuck peter - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: Peter Lin [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? the default Perm is 64M. If LeeAnn's webapp have a large number of classes, maybe there's too many; therefore forcing the VM to resize the perm. IIRC (haven't looked at core HotSpot code in detail since 1.4.1 days), the virtual address boundary between perm and the other generations is fixed at JVM initialization and never moves. The perm gen does go through GC just like the other gens, but no memory ever moves between the perm and any other space, and the perm gen maximum virtual address range never changes. The HotSpot JVM allocates, but does not commit, all of the -Xmx space at startup in one contiguous block (although there were rumors of that changing in later JVMs). Only the -Xms portion is committed, split up into three regions - young, old, and perm. When a region fills and GC doesn't reduce the usage significantly, additional pages are committed in whichever region needs expanding. I think what was happening is that LeeAnn simply had more classes than would fit in 64 MB. Nothing to do with threads, just the total number of classes in all the applications added together. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
There are no limits in tomcat. http://jakarta.apache.org/tomcat/faq/memory.html -Tim LeeAnn Pultz wrote: I have a tomcat server with multiple copies of our web application running. We are running into what seems to be a hard-coded limit on the number of threads available to the Tomcat application. I have added code to our servlet class which prints out the number of active threads whenever I initialize the servlet. On Red Hat Linux boxes, whenever we start up enough copies of the servlet to hit 100 active threads, we reach an OutOfMemoryError - regardless of the Xms/Xmx settings (memory profiling shows that we have plenty of memory available). On Windows XP, 50 seems to be the magic number. I have tried tweaking Xss parameters, my ulimit command in Linux shows unlimited - and I have reached the same results with Tomcat 3.3.1, 4.0.28 and 5.0.28 using Java 1.4.1 and 1.4.2. When I use WebLogic 8.1 on the same Linux box, I reach 241 active threads with no problems whatsoever (stopped my testing at that number) so it does not seem to be an o/s limitation, or a java limitation - which leads me to believe there is something in Tomcat? Is the number of threads available to tomcat hard coded somewhere? Is there a parameter or configuration setting that I can change to increase this? If this question is better off posted to the developers list, please someone, let me know :) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
I'm able to go up to 647 threads and 500 concurrent connections with 5.5.4 without any problems. chances are, it's a memory leak in the webapp. only way to know for sure is to profile the application. peter lin On 4/20/05, Tim Funk [EMAIL PROTECTED] wrote: There are no limits in tomcat. http://jakarta.apache.org/tomcat/faq/memory.html -Tim LeeAnn Pultz wrote: I have a tomcat server with multiple copies of our web application running. We are running into what seems to be a hard-coded limit on the number of threads available to the Tomcat application. I have added code to our servlet class which prints out the number of active threads whenever I initialize the servlet. On Red Hat Linux boxes, whenever we start up enough copies of the servlet to hit 100 active threads, we reach an OutOfMemoryError - regardless of the Xms/Xmx settings (memory profiling shows that we have plenty of memory available). On Windows XP, 50 seems to be the magic number. I have tried tweaking Xss parameters, my ulimit command in Linux shows unlimited - and I have reached the same results with Tomcat 3.3.1, 4.0.28 and 5.0.28 using Java 1.4.1 and 1.4.2. When I use WebLogic 8.1 on the same Linux box, I reach 241 active threads with no problems whatsoever (stopped my testing at that number) so it does not seem to be an o/s limitation, or a java limitation - which leads me to believe there is something in Tomcat? Is the number of threads available to tomcat hard coded somewhere? Is there a parameter or configuration setting that I can change to increase this? If this question is better off posted to the developers list, please someone, let me know :) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
This is not a memory leak in the application - we have hooked up a profiler to the application and watched the actual memory usage when causing this issue to happen. We have lots of memory available, are nowhere near the Xmx limit, and the machine has lots of memory available over and above the Xmx limit. In fact, I can cause the error to happen every single time, simply by hitting the first web page (login page) of the application - I don't have to log in, or do any work. All I have to do to get the error to happen is hit 18-19 different instances of the web application, watch the Active thread count go up to 100 and tip over the tomcat. We've gone through all of the suggested infomation out on the web - we've tweaked the Xms, Xmx and Xss parameters - none of these changes made any difference, since we're not actually running out of memory. I've reproduced this on 3 different machines, all running Red Hat Linux ES 3.0 - both with LD_ASSUME_KERNEL 2.4.19 and without, so the Linux threading model doesn't seem to affect the problem. At 04:16 AM 4/20/2005, you wrote: I'm able to go up to 647 threads and 500 concurrent connections with 5.5.4 without any problems. chances are, it's a memory leak in the webapp. only way to know for sure is to profile the application. peter lin On 4/20/05, Tim Funk [EMAIL PROTECTED] wrote: There are no limits in tomcat. http://jakarta.apache.org/tomcat/faq/memory.html -Tim LeeAnn Pultz wrote: I have a tomcat server with multiple copies of our web application running. We are running into what seems to be a hard-coded limit on the number of threads available to the Tomcat application. I have added code to our servlet class which prints out the number of active threads whenever I initialize the servlet. On Red Hat Linux boxes, whenever we start up enough copies of the servlet to hit 100 active threads, we reach an OutOfMemoryError - regardless of the Xms/Xmx settings (memory profiling shows that we have plenty of memory available). On Windows XP, 50 seems to be the magic number. I have tried tweaking Xss parameters, my ulimit command in Linux shows unlimited - and I have reached the same results with Tomcat 3.3.1, 4.0.28 and 5.0.28 using Java 1.4.1 and 1.4.2. When I use WebLogic 8.1 on the same Linux box, I reach 241 active threads with no problems whatsoever (stopped my testing at that number) so it does not seem to be an o/s limitation, or a java limitation - which leads me to believe there is something in Tomcat? Is the number of threads available to tomcat hard coded somewhere? Is there a parameter or configuration setting that I can change to increase this? If this question is better off posted to the developers list, please someone, let me know :) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
that sounds very odd. so if I understand correctly. this happens when you hit the login page. I assume this don't happen with a static html page? by any chance are you using jsp tags or some web framework? peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: This is not a memory leak in the application - we have hooked up a profiler to the application and watched the actual memory usage when causing this issue to happen. We have lots of memory available, are nowhere near the Xmx limit, and the machine has lots of memory available over and above the Xmx limit. In fact, I can cause the error to happen every single time, simply by hitting the first web page (login page) of the application - I don't have to log in, or do any work. All I have to do to get the error to happen is hit 18-19 different instances of the web application, watch the Active thread count go up to 100 and tip over the tomcat. We've gone through all of the suggested infomation out on the web - we've tweaked the Xms, Xmx and Xss parameters - none of these changes made any difference, since we're not actually running out of memory. I've reproduced this on 3 different machines, all running Red Hat Linux ES 3.0 - both with LD_ASSUME_KERNEL 2.4.19 and without, so the Linux threading model doesn't seem to affect the problem. At 04:16 AM 4/20/2005, you wrote: I'm able to go up to 647 threads and 500 concurrent connections with 5.5.4 without any problems. chances are, it's a memory leak in the webapp. only way to know for sure is to profile the application. peter lin On 4/20/05, Tim Funk [EMAIL PROTECTED] wrote: There are no limits in tomcat. http://jakarta.apache.org/tomcat/faq/memory.html -Tim LeeAnn Pultz wrote: I have a tomcat server with multiple copies of our web application running. We are running into what seems to be a hard-coded limit on the number of threads available to the Tomcat application. I have added code to our servlet class which prints out the number of active threads whenever I initialize the servlet. On Red Hat Linux boxes, whenever we start up enough copies of the servlet to hit 100 active threads, we reach an OutOfMemoryError - regardless of the Xms/Xmx settings (memory profiling shows that we have plenty of memory available). On Windows XP, 50 seems to be the magic number. I have tried tweaking Xss parameters, my ulimit command in Linux shows unlimited - and I have reached the same results with Tomcat 3.3.1, 4.0.28 and 5.0.28 using Java 1.4.1 and 1.4.2. When I use WebLogic 8.1 on the same Linux box, I reach 241 active threads with no problems whatsoever (stopped my testing at that number) so it does not seem to be an o/s limitation, or a java limitation - which leads me to believe there is something in Tomcat? Is the number of threads available to tomcat hard coded somewhere? Is there a parameter or configuration setting that I can change to increase this? If this question is better off posted to the developers list, please someone, let me know :) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: This is not a memory leak in the application - we have hooked up a profiler to the application and watched the actual memory usage when causing this issue to happen. We have lots of memory available, are nowhere near the Xmx limit, and the machine has lots of memory available over and above the Xmx limit. In fact, I can cause the error to happen every single time, simply by hitting the first web page (login page) of the application - I don't have to log in, or do any work. All I have to do to get the error to happen is hit 18-19 different instances of the web application, watch the Active thread count go up to 100 and tip over the tomcat. This isn't related to the minProcessors, maxProcessors acceptCount settings for your connector in your server.xml by any chance? Regards, -- Jason Bainbridge http://kde.org - [EMAIL PROTECTED] Personal Site - http://jasonbainbridge.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
We have no jsps in the application. The index.html page loads our servlet class, and I've put logging of active number of threads at the beginning of the servlet ini() and the very end of the service() methods in that servlet class. The application initializes, but we aren't doing any of the real work of rendering dynamic pages or talking to the database at this point. I get the same results whether my Xmx is 84 megs or 512 megs - we're just not using that much memory. At 08:48 AM 4/20/2005, Peter Lin wrote: that sounds very odd. so if I understand correctly. this happens when you hit the login page. I assume this don't happen with a static html page? by any chance are you using jsp tags or some web framework? peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: This is not a memory leak in the application - we have hooked up a profiler to the application and watched the actual memory usage when causing this issue to happen. We have lots of memory available, are nowhere near the Xmx limit, and the machine has lots of memory available over and above the Xmx limit. In fact, I can cause the error to happen every single time, simply by hitting the first web page (login page) of the application - I don't have to log in, or do any work. All I have to do to get the error to happen is hit 18-19 different instances of the web application, watch the Active thread count go up to 100 and tip over the tomcat. We've gone through all of the suggested infomation out on the web - we've tweaked the Xms, Xmx and Xss parameters - none of these changes made any difference, since we're not actually running out of memory. I've reproduced this on 3 different machines, all running Red Hat Linux ES 3.0 - both with LD_ASSUME_KERNEL 2.4.19 and without, so the Linux threading model doesn't seem to affect the problem. At 04:16 AM 4/20/2005, you wrote: I'm able to go up to 647 threads and 500 concurrent connections with 5.5.4 without any problems. chances are, it's a memory leak in the webapp. only way to know for sure is to profile the application. peter lin On 4/20/05, Tim Funk [EMAIL PROTECTED] wrote: There are no limits in tomcat. http://jakarta.apache.org/tomcat/faq/memory.html -Tim LeeAnn Pultz wrote: I have a tomcat server with multiple copies of our web application running. We are running into what seems to be a hard-coded limit on the number of threads available to the Tomcat application. I have added code to our servlet class which prints out the number of active threads whenever I initialize the servlet. On Red Hat Linux boxes, whenever we start up enough copies of the servlet to hit 100 active threads, we reach an OutOfMemoryError - regardless of the Xms/Xmx settings (memory profiling shows that we have plenty of memory available). On Windows XP, 50 seems to be the magic number. I have tried tweaking Xss parameters, my ulimit command in Linux shows unlimited - and I have reached the same results with Tomcat 3.3.1, 4.0.28 and 5.0.28 using Java 1.4.1 and 1.4.2. When I use WebLogic 8.1 on the same Linux box, I reach 241 active threads with no problems whatsoever (stopped my testing at that number) so it does not seem to be an o/s limitation, or a java limitation - which leads me to believe there is something in Tomcat? Is the number of threads available to tomcat hard coded somewhere? Is there a parameter or configuration setting that I can change to increase this? If this question is better off posted to the developers list, please someone, let me know :) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
No, unfortunately, changing those options doesn't seem to make any difference either. On tomcat 3, I even tried manipulating the thread pool settings (since in 4 and 5 those are handled inside tomcat). But I'm seeing the same problems on 3, 4 and 5.0 In fact, I can cause the error to happen every single time, simply by hitting the first web page (login page) of the application - I don't have to log in, or do any work. All I have to do to get the error to happen is hit 18-19 different instances of the web application, watch the Active thread count go up to 100 and tip over the tomcat. This isn't related to the minProcessors, maxProcessors acceptCount settings for your connector in your server.xml by any chance? Regards, -- Jason Bainbridge http://kde.org - [EMAIL PROTECTED] Personal Site - http://jasonbainbridge.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
that is some really odd behavior coming from Tomcat. I've probably run over 1K benchmarks on Tomcat over the last 3 years and I've yet to see that kind of behavior. Here are some random ideas to try, which I've used in the past. They may not help, but I figure it wouldn't hurt to suggest them. use jmeter to send 250 requests with 25 threads against the index page. then view the status in tomcat 5.5.4. Over time, you should see the number of threads drop to the minimum. Under normal conditions, tomcat will create enough threads to service those threads. after a few minutes, the thread count should drop. If it doesn't, it means something is keeping the connection open. if you see that, it is most likley an application bug. Another idea is to do netstat and see what the status of the connections are. It might be some of the connections are not closing. peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We have no jsps in the application. The index.html page loads our servlet class, and I've put logging of active number of threads at the beginning of the servlet ini() and the very end of the service() methods in that servlet class. The application initializes, but we aren't doing any of the real work of rendering dynamic pages or talking to the database at this point. I get the same results whether my Xmx is 84 megs or 512 megs - we're just not using that much memory. At 08:48 AM 4/20/2005, Peter Lin wrote: that sounds very odd. so if I understand correctly. this happens when you hit the login page. I assume this don't happen with a static html page? by any chance are you using jsp tags or some web framework? peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: This is not a memory leak in the application - we have hooked up a profiler to the application and watched the actual memory usage when causing this issue to happen. We have lots of memory available, are nowhere near the Xmx limit, and the machine has lots of memory available over and above the Xmx limit. In fact, I can cause the error to happen every single time, simply by hitting the first web page (login page) of the application - I don't have to log in, or do any work. All I have to do to get the error to happen is hit 18-19 different instances of the web application, watch the Active thread count go up to 100 and tip over the tomcat. We've gone through all of the suggested infomation out on the web - we've tweaked the Xms, Xmx and Xss parameters - none of these changes made any difference, since we're not actually running out of memory. I've reproduced this on 3 different machines, all running Red Hat Linux ES 3.0 - both with LD_ASSUME_KERNEL 2.4.19 and without, so the Linux threading model doesn't seem to affect the problem. At 04:16 AM 4/20/2005, you wrote: I'm able to go up to 647 threads and 500 concurrent connections with 5.5.4 without any problems. chances are, it's a memory leak in the webapp. only way to know for sure is to profile the application. peter lin On 4/20/05, Tim Funk [EMAIL PROTECTED] wrote: There are no limits in tomcat. http://jakarta.apache.org/tomcat/faq/memory.html -Tim LeeAnn Pultz wrote: I have a tomcat server with multiple copies of our web application running. We are running into what seems to be a hard-coded limit on the number of threads available to the Tomcat application. I have added code to our servlet class which prints out the number of active threads whenever I initialize the servlet. On Red Hat Linux boxes, whenever we start up enough copies of the servlet to hit 100 active threads, we reach an OutOfMemoryError - regardless of the Xms/Xmx settings (memory profiling shows that we have plenty of memory available). On Windows XP, 50 seems to be the magic number. I have tried tweaking Xss parameters, my ulimit command in Linux shows unlimited - and I have reached the same results with Tomcat 3.3.1, 4.0.28 and 5.0.28 using Java 1.4.1 and 1.4.2. When I use WebLogic 8.1 on the same Linux box, I reach 241 active threads with no problems whatsoever (stopped my testing at that number) so it does not seem to be an o/s limitation, or a java limitation - which leads me to believe there is something in Tomcat? Is the number of threads available to tomcat hard coded somewhere? Is there a parameter or configuration setting that I can change to increase this? If this question is better off posted to the developers list, please someone, let me know :) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
We did run some code that just makes threads, on the machine and it had no effect - I'll incorporate that code inside a servlet, so we can do this test against tomcat and see if it makes a difference. We just tried the same test against tomcat 5.0 using port 8080 and not using any of the mod_jk connectors, to see if that made a difference - we found that we started with more active threads by uncommenting that section of server.xml (started with 67 active threads rather than 44) and got a bit higher (died at 118 threads rather than 100 threads) so I'm taking that result to show that the actual number of threads allocated to the application remained about the same, with or without the usage of mod_jk. At 09:08 AM 4/20/2005, you wrote: that is some really odd behavior coming from Tomcat. I've probably run over 1K benchmarks on Tomcat over the last 3 years and I've yet to see that kind of behavior. Here are some random ideas to try, which I've used in the past. They may not help, but I figure it wouldn't hurt to suggest them. use jmeter to send 250 requests with 25 threads against the index page. then view the status in tomcat 5.5.4. Over time, you should see the number of threads drop to the minimum. Under normal conditions, tomcat will create enough threads to service those threads. after a few minutes, the thread count should drop. If it doesn't, it means something is keeping the connection open. if you see that, it is most likley an application bug. Another idea is to do netstat and see what the status of the connections are. It might be some of the connections are not closing. peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We have no jsps in the application. The index.html page loads our servlet class, and I've put logging of active number of threads at the beginning of the servlet ini() and the very end of the service() methods in that servlet class. The application initializes, but we aren't doing any of the real work of rendering dynamic pages or talking to the database at this point. I get the same results whether my Xmx is 84 megs or 512 megs - we're just not using that much memory. At 08:48 AM 4/20/2005, Peter Lin wrote: that sounds very odd. so if I understand correctly. this happens when you hit the login page. I assume this don't happen with a static html page? by any chance are you using jsp tags or some web framework? peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: This is not a memory leak in the application - we have hooked up a profiler to the application and watched the actual memory usage when causing this issue to happen. We have lots of memory available, are nowhere near the Xmx limit, and the machine has lots of memory available over and above the Xmx limit. In fact, I can cause the error to happen every single time, simply by hitting the first web page (login page) of the application - I don't have to log in, or do any work. All I have to do to get the error to happen is hit 18-19 different instances of the web application, watch the Active thread count go up to 100 and tip over the tomcat. We've gone through all of the suggested infomation out on the web - we've tweaked the Xms, Xmx and Xss parameters - none of these changes made any difference, since we're not actually running out of memory. I've reproduced this on 3 different machines, all running Red Hat Linux ES 3.0 - both with LD_ASSUME_KERNEL 2.4.19 and without, so the Linux threading model doesn't seem to affect the problem. At 04:16 AM 4/20/2005, you wrote: I'm able to go up to 647 threads and 500 concurrent connections with 5.5.4 without any problems. chances are, it's a memory leak in the webapp. only way to know for sure is to profile the application. peter lin On 4/20/05, Tim Funk [EMAIL PROTECTED] wrote: There are no limits in tomcat. http://jakarta.apache.org/tomcat/faq/memory.html -Tim LeeAnn Pultz wrote: I have a tomcat server with multiple copies of our web application running. We are running into what seems to be a hard-coded limit on the number of threads available to the Tomcat application. I have added code to our servlet class which prints out the number of active threads whenever I initialize the servlet. On Red Hat Linux boxes, whenever we start up enough copies of the servlet to hit 100 active threads, we reach an OutOfMemoryError - regardless of the Xms/Xmx settings (memory profiling shows that we have plenty of memory available). On Windows XP, 50 seems to be the magic number. I have tried tweaking Xss parameters, my ulimit command in Linux shows unlimited - and I have reached the same results with Tomcat 3.3.1, 4.0.28 and 5.0.28 using Java 1.4.1 and 1.4.2. When I use WebLogic 8.1 on the same Linux box, I
Re: OutOfMemoryError - 100 thread limit?
Perhaps this will trigger some ideas? The stack trace we get when we get the OutOfMemoryError is : 2005-04-20 09:55:33 StandardWrapperValve[ExtraView]: Allocate exception for servlet ExtraView javax.servlet.ServletException: Error allocating a servlet instance at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:691) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:144) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) at java.lang.Thread.run(Thread.java:536) - Root Cause - java.lang.OutOfMemoryError LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
that looks really bizzare to me. From the exception, it looks like tomcat can't create a new thread to process the request. It makes me think the previous threads that are done are being held by something for some odd reason. Normally the threads should be available again. What is the ramp up time for your tests? in other words, are you throwing 100 concurrent requests at it simultaneously, or ramping it up over a few seconds? peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: Perhaps this will trigger some ideas? The stack trace we get when we get the OutOfMemoryError is : 2005-04-20 09:55:33 StandardWrapperValve[ExtraView]: Allocate exception for servlet ExtraView javax.servlet.ServletException: Error allocating a servlet instance at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:691) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:144) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) at java.lang.Thread.run(Thread.java:536) - Root Cause - java.lang.OutOfMemoryError LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? Perhaps this will trigger some ideas? The stack trace we get when we get the OutOfMemoryError is : - Root Cause - java.lang.OutOfMemoryError It's what comes after this spot in the trace that might be really interesting... Unfortunately, OOME is somewhat of a catch-all - the JVM issues it whenever any OS-based resource is exhausted, as well as when out of heap space. Could be something simple like the limit on the number of open file descriptors being reached. It's also possible you're running out of perm gen space, which is not unusual if you have a lot of classes. Might want to turn on -verbose:gc just to see. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
This was first reported when we had multiple web applications running on one tomcat, and a user would be logged into the site and be using the application fine, and then if another user started working in a new site, they would get the OutOfMemory error when the first tried to load the page of their site. The first user would then start getting exceptions once the OutOfMemory error happened. When we reproduce the error, it doesn't seem to make a difference whether we simply click through a list of links, opening each site in a new window quickly, or whether we pause a few seconds in between opening new sites. So it doesn't necessarily *seem* to be dependent on multiple people firing up their sites at the same time, or over the course of several hours as different people use the system. We've identified that each instance of our application starts up with 3 active threads - we have a monitoring thread, the application thread and the user's current thread. When I start up tomcat, and hit the first site, I see about 44 active threads to start with - this seems to be affected by whether we have the Http10 connector commented out, whether the management port is set up etc. If I run the application through Jbuilder + tomcat, the active count doesn't count the tomcat' threads, and I start with 3 threads and increase by 3 threads for each new application I open in my browser, the first time. At 10:38 AM 4/20/2005, you wrote: that looks really bizzare to me. From the exception, it looks like tomcat can't create a new thread to process the request. It makes me think the previous threads that are done are being held by something for some odd reason. Normally the threads should be available again. What is the ramp up time for your tests? in other words, are you throwing 100 concurrent requests at it simultaneously, or ramping it up over a few seconds? peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: Perhaps this will trigger some ideas? The stack trace we get when we get the OutOfMemoryError is : 2005-04-20 09:55:33 StandardWrapperValve[ExtraView]: Allocate exception for servlet ExtraView javax.servlet.ServletException: Error allocating a servlet instance at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:691) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:144) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) at java.lang.Thread.run(Thread.java:536) - Root Cause - java.lang.OutOfMemoryError LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To
RE: OutOfMemoryError - 100 thread limit?
At 10:43 AM 4/20/2005, you wrote: From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? Perhaps this will trigger some ideas? The stack trace we get when we get the OutOfMemoryError is : - Root Cause - java.lang.OutOfMemoryError It's what comes after this spot in the trace that might be really interesting... Unfortunately, there's nothing after this :) Unfortunately, OOME is somewhat of a catch-all - the JVM issues it whenever any OS-based resource is exhausted, as well as when out of heap space. Could be something simple like the limit on the number of open file descriptors being reached. It's also possible you're running out of perm gen space, which is not unusual if you have a lot of classes. Might want to turn on -verbose:gc just to see. We did run ulimit and the system says everything is unlimited. I'm not familiar with perm gen space - if I turn on -verbose gc could you please tell me what I might be looking for? thanks! LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
Might want to turn on -verbose:gc just to see. I turned on verbose:gc and ran the test again: INFO: Server startup in 14608 ms Start Init Method. The CONFIG_FILE from web.xml is : WEB-INF/configuration/Configuration.properties Using Configuration FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site1/WEB-INF/configuration/Configuration.properties Successfully created a SesameConfig object from FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site1/WEB-INF/configuration/Configuration.properties SesameProjectDir: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site1/WEB-INF Attempting to Open Log File: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site1/WEB-INF/logs/EVJ.log Successfully finished init in SesameServlet. [GC 20785K-9728K(129728K), 0.0274860 secs] Threads Active : 39 [Full GC 15980K-10340K(129728K), 0.1632550 secs] [GC 22085K-11723K(129728K), 0.0286990 secs] Threads Active : 41 Threads Active : 41 Threads Active : 41 Start Init Method. The CONFIG_FILE from web.xml is : WEB-INF/configuration/Configuration.properties Using Configuration FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site2/WEB-INF/configuration/Configuration.properties Successfully created a SesameConfig object from FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site2/WEB-INF/configuration/Configuration.properties SesameProjectDir: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site2/WEB-INF Attempting to Open Log File: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site2/WEB-INF/logs/EVJ.log Successfully finished init in SesameServlet. [Full GC 23440K-12707K(129728K), 0.2036440 secs] [GC 24483K-13950K(129728K), 0.0181640 secs] Start Init Method. ... more like this Successfully finished init in SesameServlet. [GC 54635K-43725K(129792K), 0.0247320 secs] [GC 55562K-44967K(129792K), 0.0167680 secs] [Full GC 45254K-44666K(129792K), 0.6926680 secs] Threads Active : 93 Threads Active : 93 Start Init Method. The CONFIG_FILE from web.xml is : WEB-INF/configuration/Configuration.properties Using Configuration FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF/configuration/Configuration.properties Successfully created a SesameConfig object from FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF/configuration/Configuration.properties SesameProjectDir: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF Attempting to Open Log File: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site18/WEB-INF/logs/EVJ.log Successfully finished init in SesameServlet. [GC 56506K-45573K(129792K), 0.0164210 secs] [GC 57412K-46857K(129792K), 0.0237680 secs] Threads Active : 96 Threads Active : 96 Start Init Method. The CONFIG_FILE from web.xml is : WEB-INF/configuration/Configuration.properties Using Configuration FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF/configuration/Configuration.properties Successfully created a SesameConfig object from FILE: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF/configuration/Configuration.properties SesameProjectDir: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF Attempting to Open Log File: /usr/local/extraview/stage2/jakarta-tomcat-5.0.28/webapps/site19/WEB-INF/logs/EVJ.log Successfully finished init in SesameServlet. [Full GC 52758K-47299K(129792K), 0.6463800 secs] [Full GC 47308K-47304K(129792K), 0.6400790 secs] [Full GC 47304K-47304K(129792K), 0.6315800 secs] [Full GC 47304K-47042K(129792K), 0.6941910 secs] java.lang.OutOfMemoryError java.lang.OutOfMemoryError Threads Active : 94 Threads Active : 94 [Full GC 48357K-47096K(129792K), 0.6358540 secs] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: This was first reported when we had multiple web applications running on one tomcat, and a user would be logged into the site and be using the application fine, and then if another user started working in a new site, they would get the OutOfMemory error when the first tried to load the page of their site. The first user would then start getting exceptions once the OutOfMemory error happened. Okay silly question time... How are you setting -Xms and -Xmx ? Are you sure they are being picked up by Tomcat? Regards, -- Jason Bainbridge http://kde.org - [EMAIL PROTECTED] Personal Site - http://jasonbainbridge.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
the number of threads seems odd. unless your webapp creates a bunch of threads using ServletContextListener, the total thread count should be between 10-15. I have a webapp which creates 10 threads on startup. When tomcat is done and my webapp is loaded, I have a total of 38. if your webapp is creating threads at start, what are they doing? peter lin Threads Active : 94 Threads Active : 94 [Full GC 48357K-47096K(129792K), 0.6358540 secs] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError - 100 thread limit? I turned on verbose:gc and ran the test again: [Full GC 48357K-47096K(129792K), 0.6358540 secs] This shows you've given the JVM only 128 MB to work with; looks like you really are out of heap space. You probably want to increase the maximum (-Xmx) considerably. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs] [GC 57936K-47554K(129792K), 0.0238210 secs] [Full GC 49312K-47441K(129792K), 0.6546550 secs] [Full GC 47732K-47129K(129792K), 0.7706500 secs] [Full GC 48310K-47256K(129792K), 0.6425030 secs] [Full GC 47276K-47263K(129792K), 0.6445770 secs] [Full GC 47263K-47263K(129792K), 0.6368020 secs] [Full GC 47263K-47241K(129792K), 0.7775600 secs] java.lang.OutOfMemoryError [Full GC 47295K-47242K(129792K), 0.6301470 secs] [Full GC 47248K-47245K(129792K), 0.6451800 secs] [Full GC 47257K-47250K(129792K), 0.6454320 secs] [Full GC 47271K-47262K(129792K), 0.7056980 secs] java.lang.OutOfMemoryError java.lang.OutOfMemoryError So it looks like it isn't just a thread issue. If I fire up the simple servlet 18 times, I don't get the error. In fact, I can then go on to fire up our main servlet another 17-18 times before we get the OutOfMemory error - this time at 844 threads before it dies. There must be something else going on within our application that is consuming something that the simple servlet doesn't. If anyone has any suggestions on other things that I could check for ? Someone said OOM errors can mean out of file descriptors as well? thanks for all your help so far - I'm really hoping we can figure this out! At 11:20 AM 4/20/2005, you wrote: the number of threads seems odd. unless your webapp creates a bunch of threads using ServletContextListener, the total thread count should be between 10-15. I have a webapp which creates 10 threads on startup. When tomcat is done and my webapp is loaded, I have a total of 38. if your webapp is creating threads at start, what are they doing? peter lin Threads Active : 94 Threads Active : 94 [Full GC 48357K-47096K(129792K), 0.6358540 secs] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? [Full GC 47271K-47262K(129792K), 0.7056980 secs] If anyone has any suggestions on other things that I could check for ? It still looks like your heap size is too small - 128MB isn't much these days. Assuming you're using a Sun 1.4 JVM or later, try specifying -XX:+PrintGCDetails on the command line to see data about the individual generations of the heap to see which one is becoming exhausted. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
Thanks - I'll try that - we do have CATALINA_OPTS set in catalina.sh of -Xms128m and -Xmx512m. At 02:12 PM 4/20/2005, you wrote: From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? [Full GC 47271K-47262K(129792K), 0.7056980 secs] If anyone has any suggestions on other things that I could check for ? It still looks like your heap size is too small - 128MB isn't much these days. Assuming you're using a Sun 1.4 JVM or later, try specifying -XX:+PrintGCDetails on the command line to see data about the individual generations of the heap to see which one is becoming exhausted. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
After you've started your application, do a ps ax|grep java and see if that value is really making it into the command line; I'm doubting it is. To set these options, I added a the following the the .bash_profile of my tomcat user: export JAVA_OPTS=-Xmx1000m -ryan -Original Message- From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 20, 2005 4:35 PM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? Thanks - I'll try that - we do have CATALINA_OPTS set in catalina.sh of -Xms128m and -Xmx512m. At 02:12 PM 4/20/2005, you wrote: From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? [Full GC 47271K-47262K(129792K), 0.7056980 secs] If anyone has any suggestions on other things that I could check for ? It still looks like your heap size is too small - 128MB isn't much these days. Assuming you're using a Sun 1.4 JVM or later, try specifying -XX:+PrintGCDetails on the command line to see data about the individual generations of the heap to see which one is becoming exhausted. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError - 100 thread limit?
The memory parameters as well as the other CATALINA_OPTS that we set do appear on the process: [EMAIL PROTECTED] webapps]$ ps ax|grep stage2_java 866 pts/1R 0:10 /home/sesame/stage2_java/bin/java -verbose:gc -server -Xms128m -Xmx512m -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/local/extraview/stage2/tomcat/common/endorsed -classpath /home/sesame/stage2_java/lib/tools.jar:/usr/local/extraview/stage2/tomcat/bin/bootstrap.jar:/usr/local/extraview/ At 03:00 PM 4/20/2005, you wrote: After you've started your application, do a ps ax|grep java and see if that value is really making it into the command line; I'm doubting it is. To set these options, I added a the following the the .bash_profile of my tomcat user: export JAVA_OPTS=-Xmx1000m -ryan -Original Message- From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 20, 2005 4:35 PM To: Tomcat Users List Subject: RE: OutOfMemoryError - 100 thread limit? Thanks - I'll try that - we do have CATALINA_OPTS set in catalina.sh of -Xms128m and -Xmx512m. At 02:12 PM 4/20/2005, you wrote: From: LeeAnn Pultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError - 100 thread limit? [Full GC 47271K-47262K(129792K), 0.7056980 secs] If anyone has any suggestions on other things that I could check for ? It still looks like your heap size is too small - 128MB isn't much these days. Assuming you're using a Sun 1.4 JVM or later, try specifying -XX:+PrintGCDetails on the command line to see data about the individual generations of the heap to see which one is becoming exhausted. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
if your application is thread heavy, I would recommend changing it so that one thread can manage several processes. creating 800+ threads is going to hit a scalability and reliability wall very quickly. In general, you want to make sure the number of threads remain constant under constant load. If it doesn't the server will eventually crash as you currently see. I'm guessing the application is using threads to handle async processes, which is good, but spawning new threads with every request isn't going to work. hope that helps peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs] [GC 57936K-47554K(129792K), 0.0238210 secs] [Full GC 49312K-47441K(129792K), 0.6546550 secs] [Full GC 47732K-47129K(129792K), 0.7706500 secs] [Full GC 48310K-47256K(129792K), 0.6425030 secs] [Full GC 47276K-47263K(129792K), 0.6445770 secs] [Full GC 47263K-47263K(129792K), 0.6368020 secs] [Full GC 47263K-47241K(129792K), 0.7775600 secs] java.lang.OutOfMemoryError [Full GC 47295K-47242K(129792K), 0.6301470 secs] [Full GC 47248K-47245K(129792K), 0.6451800 secs] [Full GC 47257K-47250K(129792K), 0.6454320 secs] [Full GC 47271K-47262K(129792K), 0.7056980 secs] java.lang.OutOfMemoryError java.lang.OutOfMemoryError So it looks like it isn't just a thread issue. If I fire up the simple servlet 18 times, I don't get the error. In fact, I can then go on to fire up our main servlet another 17-18 times before we get the OutOfMemory error - this time at 844 threads before it dies. There must be something else going on within our application that is consuming something that the simple servlet doesn't. If anyone has any suggestions on other things that I could check for ? Someone said OOM errors can mean out of file descriptors as well? thanks for all your help so far - I'm really hoping we can figure this out! - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
I think we've gotten things a bit confused :) The application itself is not thread intensive - our original tests show that each separate copy of the web application started up 3 active threads. Further testing, when we added in debugging code that would spin up dozens of threads just to see if we could cause the tomcat to get the error if we started one application up and had tons of threads, showed that we could get 800+ threads with no problems (no outofmemory errors) and that the problem does not in fact seem to be specifically related to the number of threads the application creates. In fact, regardless of whether each instance of the application spins up 3 threads (real situation) or 10+ (debug code only) threads, we still get the out of memory error not when we hit a certain number of threads, but actually when we hit a certain number of servlets having been initialized. Testing a very simple servlet, I can start up 25 instances of the servlet with no OOM error. Testing our application's servlet, I can only start up 17-18 instances of the servlet before getting an OOM error. At this point I am going through our standard application servlet, trying to isolate what work it does that may cause a problem when we do it 17 or 18 times on one installation of tomcat. At 05:16 PM 4/20/2005, you wrote: if your application is thread heavy, I would recommend changing it so that one thread can manage several processes. creating 800+ threads is going to hit a scalability and reliability wall very quickly. In general, you want to make sure the number of threads remain constant under constant load. If it doesn't the server will eventually crash as you currently see. I'm guessing the application is using threads to handle async processes, which is good, but spawning new threads with every request isn't going to work. hope that helps peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs] [GC 57936K-47554K(129792K), 0.0238210 secs] [Full GC 49312K-47441K(129792K), 0.6546550 secs] [Full GC 47732K-47129K(129792K), 0.7706500 secs] [Full GC 48310K-47256K(129792K), 0.6425030 secs] [Full GC 47276K-47263K(129792K), 0.6445770 secs] [Full GC 47263K-47263K(129792K), 0.6368020 secs] [Full GC 47263K-47241K(129792K), 0.7775600 secs] java.lang.OutOfMemoryError [Full GC 47295K-47242K(129792K), 0.6301470 secs] [Full GC 47248K-47245K(129792K), 0.6451800 secs] [Full GC 47257K-47250K(129792K), 0.6454320 secs] [Full GC 47271K-47262K(129792K), 0.7056980 secs] java.lang.OutOfMemoryError java.lang.OutOfMemoryError So it looks like it isn't just a thread issue. If I fire up the simple servlet 18 times, I don't get the error. In fact, I can then go on to fire up our main servlet another 17-18 times before we get the OutOfMemory error - this time at 844 threads before it dies. There must be something else going on within our application that is consuming something that the simple servlet doesn't. If anyone has any suggestions on other things that I could check for ? Someone said OOM errors can mean out of file descriptors as well? thanks for all your help so far - I'm really hoping we can figure this out! - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
ahh ok. .. my confusion. back to the problem you see. I've tested tomcat with 30-40 threads without any problems in the past. Even with heavy weight JSTL tags in JSP's, I'm able to go up to 50 threads with tomcat4.1. I'll try hitting tomcat's status servlet with 50 threads tonight and see what happens. peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: I think we've gotten things a bit confused :) The application itself is not thread intensive - our original tests show that each separate copy of the web application started up 3 active threads. Further testing, when we added in debugging code that would spin up dozens of threads just to see if we could cause the tomcat to get the error if we started one application up and had tons of threads, showed that we could get 800+ threads with no problems (no outofmemory errors) and that the problem does not in fact seem to be specifically related to the number of threads the application creates. In fact, regardless of whether each instance of the application spins up 3 threads (real situation) or 10+ (debug code only) threads, we still get the out of memory error not when we hit a certain number of threads, but actually when we hit a certain number of servlets having been initialized. Testing a very simple servlet, I can start up 25 instances of the servlet with no OOM error. Testing our application's servlet, I can only start up 17-18 instances of the servlet before getting an OOM error. At this point I am going through our standard application servlet, trying to isolate what work it does that may cause a problem when we do it 17 or 18 times on one installation of tomcat. At 05:16 PM 4/20/2005, you wrote: if your application is thread heavy, I would recommend changing it so that one thread can manage several processes. creating 800+ threads is going to hit a scalability and reliability wall very quickly. In general, you want to make sure the number of threads remain constant under constant load. If it doesn't the server will eventually crash as you currently see. I'm guessing the application is using threads to handle async processes, which is good, but spawning new threads with every request isn't going to work. hope that helps peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs] [GC 57936K-47554K(129792K), 0.0238210 secs] [Full GC 49312K-47441K(129792K), 0.6546550 secs] [Full GC 47732K-47129K(129792K), 0.7706500 secs] [Full GC 48310K-47256K(129792K), 0.6425030 secs] [Full GC 47276K-47263K(129792K), 0.6445770 secs] [Full GC 47263K-47263K(129792K), 0.6368020 secs] [Full GC 47263K-47241K(129792K), 0.7775600 secs] java.lang.OutOfMemoryError [Full GC 47295K-47242K(129792K), 0.6301470 secs] [Full GC 47248K-47245K(129792K), 0.6451800 secs] [Full GC 47257K-47250K(129792K), 0.6454320 secs] [Full GC 47271K-47262K(129792K), 0.7056980 secs] java.lang.OutOfMemoryError java.lang.OutOfMemoryError So it looks like it isn't just a thread issue. If I fire up the simple servlet 18 times, I don't get the error. In fact, I can then go on to fire up our main servlet another 17-18 times before we get the OutOfMemory error - this time at 844 threads before it dies. There must be something else going on within our application that is consuming something that the simple servlet doesn't. If anyone has any suggestions on other things that I could check for ? Someone said OOM errors can mean out of file descriptors as well? thanks for all your help so far - I'm really hoping we can figure this out! - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] LeeAnn Pultz ExtraView Corporation [EMAIL PROTECTED] 831-461-7100 x115 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - 100 thread limit?
when you say 50 threads, do you mean 50 separate web applications? My concern right now is that we seem to be limited to 17-18 copies of our web app on a particular installation of tomcat. And that is just hitting the login page - not doing any work. Practically, for applications with many users, we may be limited to 7-10 copies of our application on a tomcat. Maybe I'm expecting too much? Perhaps that's a reasonable number of concurrent webapps that can be run on a tomcat? But if that is the case, it would be good to know what the limiting factor is, if it's not number of threads or amount of memory. thanks for much for your help! At 05:48 PM 4/20/2005, you wrote: ahh ok. .. my confusion. back to the problem you see. I've tested tomcat with 30-40 threads without any problems in the past. Even with heavy weight JSTL tags in JSP's, I'm able to go up to 50 threads with tomcat4.1. I'll try hitting tomcat's status servlet with 50 threads tonight and see what happens. peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: I think we've gotten things a bit confused :) The application itself is not thread intensive - our original tests show that each separate copy of the web application started up 3 active threads. Further testing, when we added in debugging code that would spin up dozens of threads just to see if we could cause the tomcat to get the error if we started one application up and had tons of threads, showed that we could get 800+ threads with no problems (no outofmemory errors) and that the problem does not in fact seem to be specifically related to the number of threads the application creates. In fact, regardless of whether each instance of the application spins up 3 threads (real situation) or 10+ (debug code only) threads, we still get the out of memory error not when we hit a certain number of threads, but actually when we hit a certain number of servlets having been initialized. Testing a very simple servlet, I can start up 25 instances of the servlet with no OOM error. Testing our application's servlet, I can only start up 17-18 instances of the servlet before getting an OOM error. At this point I am going through our standard application servlet, trying to isolate what work it does that may cause a problem when we do it 17 or 18 times on one installation of tomcat. At 05:16 PM 4/20/2005, you wrote: if your application is thread heavy, I would recommend changing it so that one thread can manage several processes. creating 800+ threads is going to hit a scalability and reliability wall very quickly. In general, you want to make sure the number of threads remain constant under constant load. If it doesn't the server will eventually crash as you currently see. I'm guessing the application is using threads to handle async processes, which is good, but spawning new threads with every request isn't going to work. hope that helps peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs] [GC 57936K-47554K(129792K), 0.0238210 secs] [Full GC 49312K-47441K(129792K), 0.6546550 secs] [Full GC 47732K-47129K(129792K), 0.7706500 secs] [Full GC 48310K-47256K(129792K), 0.6425030 secs] [Full GC 47276K-47263K(129792K), 0.6445770 secs] [Full GC 47263K-47263K(129792K), 0.6368020 secs] [Full GC 47263K-47241K(129792K), 0.7775600 secs] java.lang.OutOfMemoryError [Full GC 47295K-47242K(129792K), 0.6301470 secs] [Full GC 47248K-47245K(129792K), 0.6451800 secs] [Full GC 47257K-47250K(129792K), 0.6454320 secs] [Full GC 47271K-47262K(129792K), 0.7056980 secs] java.lang.OutOfMemoryError java.lang.OutOfMemoryError So it looks like it isn't just a thread issue. If I fire up the simple servlet 18 times, I don't get the error. In fact, I can then go on to fire up our main servlet another 17-18 times before we get the OutOfMemory error - this time at 844 threads before it dies. There must be something else going on within our application that is consuming something that the simple servlet doesn't. If anyone has any suggestions on other things that I could check for ? Someone said OOM errors can mean out of file descriptors as well? thanks for all your help so far - I'm really hoping we can figure this out!
Re: OutOfMemoryError - 100 thread limit?
I'll try loading up 18 version of my webapp and see if tomcat blows up. should know in a hour or so peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: when you say 50 threads, do you mean 50 separate web applications? My concern right now is that we seem to be limited to 17-18 copies of our web app on a particular installation of tomcat. And that is just hitting the login page - not doing any work. Practically, for applications with many users, we may be limited to 7-10 copies of our application on a tomcat. Maybe I'm expecting too much? Perhaps that's a reasonable number of concurrent webapps that can be run on a tomcat? But if that is the case, it would be good to know what the limiting factor is, if it's not number of threads or amount of memory. thanks for much for your help! At 05:48 PM 4/20/2005, you wrote: ahh ok. .. my confusion. back to the problem you see. I've tested tomcat with 30-40 threads without any problems in the past. Even with heavy weight JSTL tags in JSP's, I'm able to go up to 50 threads with tomcat4.1. I'll try hitting tomcat's status servlet with 50 threads tonight and see what happens. peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: I think we've gotten things a bit confused :) The application itself is not thread intensive - our original tests show that each separate copy of the web application started up 3 active threads. Further testing, when we added in debugging code that would spin up dozens of threads just to see if we could cause the tomcat to get the error if we started one application up and had tons of threads, showed that we could get 800+ threads with no problems (no outofmemory errors) and that the problem does not in fact seem to be specifically related to the number of threads the application creates. In fact, regardless of whether each instance of the application spins up 3 threads (real situation) or 10+ (debug code only) threads, we still get the out of memory error not when we hit a certain number of threads, but actually when we hit a certain number of servlets having been initialized. Testing a very simple servlet, I can start up 25 instances of the servlet with no OOM error. Testing our application's servlet, I can only start up 17-18 instances of the servlet before getting an OOM error. At this point I am going through our standard application servlet, trying to isolate what work it does that may cause a problem when we do it 17 or 18 times on one installation of tomcat. At 05:16 PM 4/20/2005, you wrote: if your application is thread heavy, I would recommend changing it so that one thread can manage several processes. creating 800+ threads is going to hit a scalability and reliability wall very quickly. In general, you want to make sure the number of threads remain constant under constant load. If it doesn't the server will eventually crash as you currently see. I'm guessing the application is using threads to handle async processes, which is good, but spawning new threads with every request isn't going to work. hope that helps peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs] [GC 57936K-47554K(129792K), 0.0238210 secs] [Full GC 49312K-47441K(129792K), 0.6546550 secs] [Full GC 47732K-47129K(129792K), 0.7706500 secs] [Full GC 48310K-47256K(129792K), 0.6425030 secs] [Full GC 47276K-47263K(129792K), 0.6445770 secs] [Full GC 47263K-47263K(129792K), 0.6368020 secs] [Full GC 47263K-47241K(129792K), 0.7775600 secs] java.lang.OutOfMemoryError [Full GC 47295K-47242K(129792K), 0.6301470 secs] [Full GC 47248K-47245K(129792K), 0.6451800 secs] [Full GC 47257K-47250K(129792K), 0.6454320 secs] [Full GC 47271K-47262K(129792K), 0.7056980 secs] java.lang.OutOfMemoryError java.lang.OutOfMemoryError So it looks like it isn't just a thread issue. If I fire up the simple servlet 18 times, I don't get the error. In fact, I can then go on to fire up our main servlet another 17-18 times before we get the OutOfMemory error - this time at 844 threads before it dies. There must
Re: OutOfMemoryError - 100 thread limit?
ok, I just tried starting tomcat with the default heap size and 18 instances of my webapp and it got OOME just as you see. If I set my heap to -Xms256m -Xmx512m I'm able to load the 18 webapps just fine. the total memory used after all the webapps are loaded is 152Megs. the total number of webapps is 25 webapps including the default tomcat webapps and a few more. so basically you'll need to set the heap larger enough and it should work. If i do some math, we can calculate a rough estimate for the min heap (154 - 64)/25 = 3.6megs so basically, you'll need to multiply the number of webapps by 4 megs to figure out the heap setting. peter lin On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: when you say 50 threads, do you mean 50 separate web applications? My concern right now is that we seem to be limited to 17-18 copies of our web app on a particular installation of tomcat. And that is just hitting the login page - not doing any work. Practically, for applications with many users, we may be limited to 7-10 copies of our application on a tomcat. Maybe I'm expecting too much? Perhaps that's a reasonable number of concurrent webapps that can be run on a tomcat? But if that is the case, it would be good to know what the limiting factor is, if it's not number of threads or amount of memory. thanks for much for your help! At 05:48 PM 4/20/2005, you wrote: ahh ok. .. my confusion. back to the problem you see. I've tested tomcat with 30-40 threads without any problems in the past. Even with heavy weight JSTL tags in JSP's, I'm able to go up to 50 threads with tomcat4.1. I'll try hitting tomcat's status servlet with 50 threads tonight and see what happens. peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: I think we've gotten things a bit confused :) The application itself is not thread intensive - our original tests show that each separate copy of the web application started up 3 active threads. Further testing, when we added in debugging code that would spin up dozens of threads just to see if we could cause the tomcat to get the error if we started one application up and had tons of threads, showed that we could get 800+ threads with no problems (no outofmemory errors) and that the problem does not in fact seem to be specifically related to the number of threads the application creates. In fact, regardless of whether each instance of the application spins up 3 threads (real situation) or 10+ (debug code only) threads, we still get the out of memory error not when we hit a certain number of threads, but actually when we hit a certain number of servlets having been initialized. Testing a very simple servlet, I can start up 25 instances of the servlet with no OOM error. Testing our application's servlet, I can only start up 17-18 instances of the servlet before getting an OOM error. At this point I am going through our standard application servlet, trying to isolate what work it does that may cause a problem when we do it 17 or 18 times on one installation of tomcat. At 05:16 PM 4/20/2005, you wrote: if your application is thread heavy, I would recommend changing it so that one thread can manage several processes. creating 800+ threads is going to hit a scalability and reliability wall very quickly. In general, you want to make sure the number of threads remain constant under constant load. If it doesn't the server will eventually crash as you currently see. I'm guessing the application is using threads to handle async processes, which is good, but spawning new threads with every request isn't going to work. hope that helps peter On 4/20/05, LeeAnn Pultz [EMAIL PROTECTED] wrote: We tried a new test :) We added code that spins up 8 threads inside just a plain servlet that doesn't do anything else. When I fire up 10 different instances of that servlet, I get hundreds of threads printing out as being active - and no out of memory errors. So we added the same code to our (slightly more complex) application servlet, and did the same thing. Now the reported number of threads is much higher, but I still get an OutOfMemory error at the same number of new instances fired up (between 17-18 sites started up). Active Threads : WHEN STARTING INIT() 617 [GC 57193K-46096K(129792K), 0.0353880 secs] [GC 57936K-47554K(129792K), 0.0238210 secs] [Full GC 49312K-47441K(129792K), 0.6546550 secs] [Full GC 47732K-47129K(129792K), 0.7706500 secs] [Full GC 48310K-47256K(129792K), 0.6425030 secs] [Full GC 47276K-47263K(129792K), 0.6445770 secs] [Full GC 47263K-47263K(129792K), 0.6368020 secs] [Full GC 47263K-47241K(129792K), 0.7775600 secs] java.lang.OutOfMemoryError [Full GC 47295K-47242K(129792K), 0.6301470 secs]
RE: OutOfMemoryError: unable to create new native thread
Hi. No problem! :) Reduced Maintenance is definitely one reason. Another reason is that I generally share a connector on the same port/protocol - and place all of these in a Service / The other connectors are all for HTTPS sites - each running on the same port but bound to different Ips. I also have a management tool that abstracts all the conf entries - and doing this in one tomcat instance is easier than with multiple! Question: Having multiple tomcat instances on the same server - would this not increase the memory usage - due to multiple JVMs? Thanks Carl -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: Thursday, October 28, 2004 6:56 PM To: Tomcat Users List Subject: RE: OutOfMemoryError: unable to create new native thread Hi, Apparently Windows NT/2000 defaults to a 1mb virtual thread stack size per thread unless otherwise stated. In addition a maximum of 2gb of thread stack space per process is allowed by Windows (not sure if this can be changed or not!). Thus the tomcat JVM could only have 2000 odd concurrent threads - and on a server that hosts 50 sites, of which 20 are fairly busy this could be a problem. Thanks for posting your findings and solutions. These type of posts make the best archival materials, assuming people bother to search the archives. Just out of curiosity, why are you running all the sites on one Tomcat instance? Reduced maintenance? I'm not arguing against it, it clearly works fine, and I do similar things myself. Just curious for your reasoning. Yoav This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError: unable to create new native thread
Hi, Question: Having multiple tomcat instances on the same server - would this not increase the memory usage - due to multiple JVMs? If configured correctly, the total should be just about the same. It might be a bit higher or lower depending on the semantics of your webapp (% in old generation vs. % new, from a GC perspective). But it shouldn't be a big change from a total memory consumption standpoint. And on the plus side, you'd let each JVM tune itself optimally for its webapp(s), something that's virtually impossible to do with more than a couple of webapps in one JVM. Yoav This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError: unable to create new native thread
Hi, - Actually ran out of memory - Hit max # of threads allowed to run (OS dependant?) This is the most likely cause in your case, and it's OS-dependent as you surmised. On *nix this is adjusted via the ulimit (-n) command. But on Windows 2000 Server, I'm not sure off the top of my head. Google? Yoav This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError: unable to create new native thread
Hi Yoav. Thanks for the response. I believe I have found the issue. Apparently Windows NT/2000 defaults to a 1mb virtual thread stack size per thread unless otherwise stated. In addition a maximum of 2gb of thread stack space per process is allowed by Windows (not sure if this can be changed or not!). Thus the tomcat JVM could only have 2000 odd concurrent threads - and on a server that hosts 50 sites, of which 20 are fairly busy this could be a problem. Thus I reduced the thread stack size to 512k (via -Xss512k). In addition there were about 12 connectors in tomcat where the minSpareThreads were set too high - so set them back to default of 4 (from between 50 and 100). Hectic. Now we see how she bahaves! Regards, Carl -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: Thursday, October 28, 2004 5:38 PM To: Tomcat Users List Subject: RE: OutOfMemoryError: unable to create new native thread Hi, - Actually ran out of memory - Hit max # of threads allowed to run (OS dependant?) This is the most likely cause in your case, and it's OS-dependent as you surmised. On *nix this is adjusted via the ulimit (-n) command. But on Windows 2000 Server, I'm not sure off the top of my head. Google? Yoav This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError: unable to create new native thread
Hi, Apparently Windows NT/2000 defaults to a 1mb virtual thread stack size per thread unless otherwise stated. In addition a maximum of 2gb of thread stack space per process is allowed by Windows (not sure if this can be changed or not!). Thus the tomcat JVM could only have 2000 odd concurrent threads - and on a server that hosts 50 sites, of which 20 are fairly busy this could be a problem. Thanks for posting your findings and solutions. These type of posts make the best archival materials, assuming people bother to search the archives. Just out of curiosity, why are you running all the sites on one Tomcat instance? Reduced maintenance? I'm not arguing against it, it clearly works fine, and I do similar things myself. Just curious for your reasoning. Yoav This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError after 5.0.24 update
Stefan Proels wrote: Sorry, I wasn't precise enough here. I'm not statically precompiling JSPs that way. Sometimes we do need the feature to update a JSPs while the application is running. I'm running a simple script which invokes every JSP with jsp_precompile=true instead and since this script already ran I wonder what CompilerThread1 still has to do. I've had several more crashes, all with an OutOfMemoryError in CompilerThread1 without the Java process actually running out of heap. I finally went back to 5.0.19 for now. I still hope there is a possibility to fix this problem somehow, because 5.0.24 contains several bug fixes I was waiting for and thus I'd like to update. You can disable the background reloading thread, I think. Use reloading=false and development=false in web.xml. -- x Rémy Maucherat Developer Consultant JBoss Group (Europe) Sà RL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError after 5.0.24 update
On Monday 10 May 2004 17:26, Remy Maucherat wrote: Stefan Proels wrote: Dear all, after updating our server to Tomcat 5.0.24 last night I got 2 crashes with the last words Exception in thread CompilerThread1 java.lang.OutOfMemoryError: requested 32760 bytes for ChunkPool::allocate. Out of swap space? from Tomcat. Now the interesting is that according to our GC logs Java did not really run out of memory. Also I checked the system's state some minutes before the last crash and I remember the java process having allocated about 270 MB and as I set -Xmx1024m this also makes be believe that it did not really run of the memory. The pointer to the swap space also doesn't seem to lead into the right direction since the machine has 4 GB physical memory and another 4 GB swap space and nearly no swap space is actually in use. So my question is, is there any new configuration parameter I my have overlooked since 5.0.19 or has the default value of some related parameter changed which could cause this error? What makes the problem even more mysterious is the fact that both exceptions occurred in CompilerThread1 but since all our JSPs have been compiled already and none of them has changed, I wonder why this thread had anything to do at all. You can (= should) disable the compiler thread IMO. If you're precompiling every JSPs to servlets (with mappings in web.xml), you should leave Jasper in development mode, and it won't spawn the thread. Sorry, I wasn't precise enough here. I'm not statically precompiling JSPs that way. Sometimes we do need the feature to update a JSPs while the application is running. I'm running a simple script which invokes every JSP with jsp_precompile=true instead and since this script already ran I wonder what CompilerThread1 still has to do. I've had several more crashes, all with an OutOfMemoryError in CompilerThread1 without the Java process actually running out of heap. I finally went back to 5.0.19 for now. I still hope there is a possibility to fix this problem somehow, because 5.0.24 contains several bug fixes I was waiting for and thus I'd like to update. Best regards, Stefan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError after 5.0.24 update
Stefan Proels wrote: Dear all, after updating our server to Tomcat 5.0.24 last night I got 2 crashes with the last words Exception in thread CompilerThread1 java.lang.OutOfMemoryError: requested 32760 bytes for ChunkPool::allocate. Out of swap space? from Tomcat. Now the interesting is that according to our GC logs Java did not really run out of memory. Also I checked the system's state some minutes before the last crash and I remember the java process having allocated about 270 MB and as I set -Xmx1024m this also makes be believe that it did not really run of the memory. The pointer to the swap space also doesn't seem to lead into the right direction since the machine has 4 GB physical memory and another 4 GB swap space and nearly no swap space is actually in use. So my question is, is there any new configuration parameter I my have overlooked since 5.0.19 or has the default value of some related parameter changed which could cause this error? What makes the problem even more mysterious is the fact that both exceptions occurred in CompilerThread1 but since all our JSPs have been compiled already and none of them has changed, I wonder why this thread had anything to do at all. You can (= should) disable the compiler thread IMO. If you're precompiling every JSPs to servlets (with mappings in web.xml), you should leave Jasper in development mode, and it won't spawn the thread. -- x Rémy Maucherat Developer Consultant JBoss Group (Europe) Sà RL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError - Excessive class loading
Sorry I forgot to mention environment details: WinXP 2000 professional, 256 MB RAM, 384 Page File, MySql 4, Tomcat 4.1.27 - Original Message - From: Chandu To: Tomcat Users List Sent: Thursday, April 01, 2004 6:49 PM Subject: OutOfMemoryError - Excessive class loading Guys, I am getting java.lang.OutOfMemoryError when I start/stop/start/stop... a specific context multiple times. I am using Optimizeit to profile my application. I found multiple class definitions are being held by tomcat of same class. Everytime the context is restarted(using the Manager application), of course, the classes will be loaded again. As developing the application, I used to start/stop a specific context so many times. However, when the restarts of context(without shutting down the tomcat instance) exceeds 30 or 40 I am getting the out of memory error. More specifically I would like to know: When a context is stopped, does all the resources will be released? This includes stopping of threads if any and unloading of all classes which are loaded during the context lifecycle. Can anyone please help me in solving this? - Chandu.
RE: OutOfMemoryError - Excessive class loading
Hi, When a context is stopped, does all the resources will be released? This includes stopping of threads if any and unloading of all classes which are loaded during the context lifecycle. No, some resources such as the classloader itself may be retained in memory. A new one will be provided for your app with new copies of your classes, that's why you're seeing multiple definitions. You can do things to help this, such as cleaning up all your references when the webapp is destroyed (a ServletContextListener's contextDestroyed and the Servlet#destroy methods are good places for this), including logging services, background threads, and the like. If you don't clean up, each webapp reload will consume more memory, including in the heap's permanent generation, eventually resulting in OutOfMemoryErrors. This is why most people don't reload webapps in production, or if they do they also restart the server periodically. Of course, all of this (and more specific details) has been discussed numerous times on this list, and is therefore available in the archives, but I've given up on hoping people would research before posting questions. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
Hi, Use jvmstat (google it) tool to analyize it. Maybe your application memory needs can break down Tomcat Regards. - Original Message - From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, March 15, 2004 8:18 AM Subject: OutOfMemoryError Hi, I am getting the following error message when I have accessed Tomcat for more than 2 days without shutting it down. I am using Tomcat 5.0.19 256 MB Ram Pentium III Win 2k Pro Mar 13, 2004 3:57:54 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren SEVERE: Exception invoking periodic operation: java.lang.OutOfMemoryError Any help on how to solve the problem would be great. Do I need to create multiple instances of the server for load balancing? Thank you, Best Regards, Uma - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
Hi, I do I control it instead of viewing my memory usage? Thank you, Best Regards, Uma - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Howdy, Although the possibility of a memory leak (sorry--unintended object retention) has not been completely ruled out (yes, I'm running the applications through a profiler), I'm quite puzzled that this problem would occur when, by all indications, memory use is nowhere near the limit. How do you know when the OutOfMemoryError occurred? Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Well, I have made an assumption in most cases. The OutOfMemoryError is logged with no timestamp to $CATALINA_HOME/logs/catalina.out. We have a script that periodically (every 15 minutes) hits the applications and notifies us of any error response (either status above 400 or the word ERROR in the content). When I get such a notification, I immediately check the log file (catalina.out) for the OutOfMemoryError, and it is always near or at the bottom of the file. I know that in and of itself is no indication of when the error actually occurs. The applications themselves are set up to use syslog (through Log4J). Sometimes the OutOfMemoryError is logged here with a timestamp, and always occurs within minutes of my notification. --- Shapira, Yoav [EMAIL PROTECTED] wrote: Howdy, Although the possibility of a memory leak (sorry--unintended object retention) has not been completely ruled out (yes, I'm running the applications through a profiler), I'm quite puzzled that this problem would occur when, by all indications, memory use is nowhere near the limit. How do you know when the OutOfMemoryError occurred? Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
A while back somebody mentioned that if you continuously redeploy a web application, the symptoms you are mentioning occur. This is because of all of the objects that are stored in the permanent generation memory. This permanent generation runs out of space, even though the new generation and old generation heaps still have room to grow. (The permanent generation's size is a fixed percentage of the total memory). I do not know if this is something that can or is fixed in the most recent builds and latest major version (5) of Tomcat. On NetWare, we have a view into the memory where we can see how big the perm gen is and what percentage of it is allocated. I do not know if such a thing exists on other platforms though. If it does, I certainly have not yet found it. Searching the archives (at http://marc.theaimsgroup.com ), it looks like it could have something to do with the classloader layout of your application: http://marc.theaimsgroup.com/?l=tomcat-userm=107046034505454w=2 Although the possibility of a memory leak (sorry--unintended object retention) has not been completely ruled out (yes, I'm running the applications through a profiler), I'm quite puzzled that this problem would occur when, by all indications, memory use is nowhere near the limit. Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Hmmm...we don't do hot deploys, but the different generations are definitely something to look at. Thanks! Dhruva --- Jeff Tulley [EMAIL PROTECTED] wrote: A while back somebody mentioned that if you continuously redeploy a web application, the symptoms you are mentioning occur. This is because of all of the objects that are stored in the permanent generation memory. This permanent generation runs out of space, even though the new generation and old generation heaps still have room to grow. (The permanent generation's size is a fixed percentage of the total memory). I do not know if this is something that can or is fixed in the most recent builds and latest major version (5) of Tomcat. On NetWare, we have a view into the memory where we can see how big the perm gen is and what percentage of it is allocated. I do not know if such a thing exists on other platforms though. If it does, I certainly have not yet found it. Searching the archives (at http://marc.theaimsgroup.com ), it looks like it could have something to do with the classloader layout of your application: http://marc.theaimsgroup.com/?l=tomcat-userm=107046034505454w=2 Although the possibility of a memory leak (sorry--unintended object retention) has not been completely ruled out (yes, I'm running the applications through a profiler), I'm quite puzzled that this problem would occur when, by all indications, memory use is nowhere near the limit. Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Wasn't there also a thread recently about a leak that can be configured away? The usage pattern to invoke the 'leak' was to load the site, then leave it alone, then load it again. It involved the worker thread pool for connections and JMX registration of said threads? The pool thought it was downsizing itself, but in fact it was leaking them away. So if you have lots of up and down loading going on, perhaps you can look into this further. -Original Message- From: Jeff Tulley [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 27, 2004 11:00 AM To: [EMAIL PROTECTED] Subject: RE: OutOfMemoryError with Seemingly Plenty of Memory A while back somebody mentioned that if you continuously redeploy a web application, the symptoms you are mentioning occur. This is because of all of the objects that are stored in the permanent generation memory. This permanent generation runs out of space, even though the new generation and old generation heaps still have room to grow. (The permanent generation's size is a fixed percentage of the total memory). I do not know if this is something that can or is fixed in the most recent builds and latest major version (5) of Tomcat. On NetWare, we have a view into the memory where we can see how big the perm gen is and what percentage of it is allocated. I do not know if such a thing exists on other platforms though. If it does, I certainly have not yet found it. Searching the archives (at http://marc.theaimsgroup.com ), it looks like it could have something to do with the classloader layout of your application: http://marc.theaimsgroup.com/?l=tomcat-userm=107046034505454w=2 Although the possibility of a memory leak (sorry--unintended object retention) has not been completely ruled out (yes, I'm running the applications through a profiler), I'm quite puzzled that this problem would occur when, by all indications, memory use is nowhere near the limit. Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError with Seemingly Plenty of Memory
Do you edit the web.xml of your web application? That will cause Tomcat to redeploy the application. We had an application that would do this during their installation (so, only one time really), but I can imagine an application that might be trying to dynamically deploy servlets and set their mappings in the web.xml. I'm guessing this is NOT your situation though, since you'd probably already be thinking along the lines of redeploys. [EMAIL PROTECTED] 1/27/04 10:52:34 AM Hmmm...we don't do hot deploys, but the different generations are definitely something to look at. Thanks! Dhruva --- Jeff Tulley [EMAIL PROTECTED] wrote: A while back somebody mentioned that if you continuously redeploy a web application, the symptoms you are mentioning occur. This is because of all of the objects that are stored in the permanent generation memory. This permanent generation runs out of space, even though the new generation and old generation heaps still have room to grow. (The permanent generation's size is a fixed percentage of the total memory). I do not know if this is something that can or is fixed in the most recent builds and latest major version (5) of Tomcat. On NetWare, we have a view into the memory where we can see how big the perm gen is and what percentage of it is allocated. I do not know if such a thing exists on other platforms though. If it does, I certainly have not yet found it. Searching the archives (at http://marc.theaimsgroup.com ), it looks like it could have something to do with the classloader layout of your application: http://marc.theaimsgroup.com/?l=tomcat-userm=107046034505454w=2 Although the possibility of a memory leak (sorry--unintended object retention) has not been completely ruled out (yes, I'm running the applications through a profiler), I'm quite puzzled that this problem would occur when, by all indications, memory use is nowhere near the limit. Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError on Tomcat-4.1.24
Hans Steinraht wrote: I'm installed Tomcat-4.1.24 on a Linux Debian pc with j2sdk1.4.1. On that there is an application running that causes Tomcat stoppping from time to time. The error I get is: Dec 2, 2003 1:44:33 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError It's not predictable when it happens, sometimes 3 weeks all seems to go right and than in 2 days Tomcat stops. Can any one point me to a direction where I have to look for. I will point you to the list archives http://marc.theaimsgroup.com/?l=tomcat-user Possible causes for OOM as well as techniques to fix them are there. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
Christopher Schultz wrote: So, moving on, what usually happens is that a low memory condition triggers a GC (usually a full GC). When the GC runs, it needs some memory to work with. If it cannot allocate memory for itself, it will die with an OOM. If the heap hasn't increased, yet, then the GC cannot allocate more objects on the heap to do it's thing. That can explain why you get an OOM when you are way under the heap size in the output of ps. Hi Christopher, I have a doubt with respect to your explanation. Is there any difference between an OOM thrown within the GC thread, and any other OOM? I mean, if an OOM is thrown within the GC, and it dies, the JVM looses its GC, so no memory can be deallocated, and the only way is to shutdown the JVM. But what happens if the OOM is thrown in any other thread? Is the JVM state preserved in this case? As said in another post, generally speaking, an OOM leaves the JVM in a bad state. But, can this state be detected by checking the state of any of the JVM threads? I guess the GC should run in a thread I could get a reference from. If I check this reference liveness, could I determine if the OOM has affected the GC? I have developed some web services that can return an important amount of data. Under high load, the server memory requirements can become huge, so I added a loop into the catalina script to restart my tomcat instance automatically, and added a filter that catches OOMs and forces a JVM shutdown, after writing some logs asking for more memory. I was wondering if I could reduce the shutdown condition to only the cases when the GC has died. Greetings, Rodrigo Ruiz - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError
Howdy, Aah. I think I see a part of the problem. First of all, the amount of memory shown by 'ps' is completely irrelevant, except that it shows how much memory that the OS has allocated to the Java process. This is allowed to be more than the max heap size set for Java. It's not completely irrelevant, but close ;) Do you have any 3rd party containers or such that live outside of the context (their JARs would be in CATALINA_HOME/server/lib or CATALINA_HOME/common/lib -- someone please confirm that the second of these two does *not* get discarded along with the rest of the context upon reload). I'll confirm that -- common classloader repositories (as well as shared and server) do not get discarded on webapp reload. They can't, as that might destroy other webapps. Yet another reason never to use these repositories, keep your webapp self-contained. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError
Howdy, Dhruva B. Reddy wrote: We get OutOfMemoryError's on Tomcat 4.1.27, running on RH 7.3, Sun JDK 1.4.1_02 (with 512MB allocated to it) The box has 1GB of RAM, 2GB of swap space, and four Xeon processors. Can you reproduce the errors in a test environment? Are you using the context reloading feature? reloadable=true or using the tomcat manager app will cause this to happen eventually. Really? Prove that please... calls Runtime.freeMemory() and .totalMemory(), it comes nowhere near using that much (we track it every five minutes by hitting a servlet that calls these methods). It could easily happen that: - You check memory, everything is fine - User makes a request for a large dataset, you load it, run out of memory. So your check is not conclusive. The stack trace suggested that the error occurred while iterating through a ResultSet, so one thing you should log are all the SQL queries you run and what they return: you probably got a large one to cause the error. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError
-Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: December 2, 2003 6:55 AM To: Tomcat Users List Subject: RE: OutOfMemoryError Howdy, Dhruva B. Reddy wrote: We get OutOfMemoryError's on Tomcat 4.1.27, running on RH 7.3, Sun JDK 1.4.1_02 (with 512MB allocated to it) The box has 1GB of RAM, 2GB of swap space, and four Xeon processors. Can you reproduce the errors in a test environment? Are you using the context reloading feature? reloadable=true or using the tomcat manager app will cause this to happen eventually. Really? Prove that please... Actually, I can't prove it in his instance, but I have proved it time and time again on ours. I'm using 4.0.4 right now though so I haven't verified 4.1.x. But, I heard that this is still the case in 4.1.x. To reproduce this problem just reload a context over and over. Depending on how much memory it uses, it will start throwing OutOfMemory exceptions long before the -mx JVM parameter value is ever reached. eg. I do a ps on linux and it shows it as using like 140M or something like that and it already starts throwing out of memory exceptions. With our applications that we're running, we can usually get away with roughly 5-10 context reloads. calls Runtime.freeMemory() and .totalMemory(), it comes nowhere near using that much (we track it every five minutes by hitting a servlet that calls these methods). It could easily happen that: - You check memory, everything is fine - User makes a request for a large dataset, you load it, run out of memory. So your check is not conclusive. The stack trace suggested that the error occurred while iterating through a ResultSet, so one thing you should log are all the SQL queries you run and what they return: you probably got a large one to cause the error. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ This communication is intended for the use of the recipient to whom it is addressed, and may contain confidential, personal, and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communications received in error, or subsequent reply, should be deleted or destroyed. --- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
Trenton, Are you using the context reloading feature? reloadable=true or using the tomcat manager app will cause this to happen eventually. Really? Prove that please... Actually, I can't prove it in his instance, but I have proved it time and time again on ours. I'm using 4.0.4 right now though so I haven't verified 4.1.x. But, I heard that this is still the case in 4.1.x. To reproduce this problem just reload a context over and over. Depending on how much memory it uses, it will start throwing OutOfMemory exceptions long before the -mx JVM parameter value is ever reached. eg. I do a ps on linux and it shows it as using like 140M or something like that and it already starts throwing out of memory exceptions. Aah. I think I see a part of the problem. First of all, the amount of memory shown by 'ps' is completely irrelevant, except that it shows how much memory that the OS has allocated to the Java process. This is allowed to be more than the max heap size set for Java. The heap is not related to the amount of memory the OS allocates, except that the OS report will always be larger than the heap report from Java. You can get an OOM even if you have the heap set to 1GB and yet the process only has 50MB used. This is because OOMs usually happen *while the GC is doing its work*. I'm not sure if there's another thread that increases the actual size of the heap for you, but if so, I'm sure it's related to the GC anyway. So, moving on, what usually happens is that a low memory condition triggers a GC (usually a full GC). When the GC runs, it needs some memory to work with. If it cannot allocate memory for itself, it will die with an OOM. If the heap hasn't increased, yet, then the GC cannot allocate more objects on the heap to do it's thing. That can explain why you get an OOM when you are way under the heap size in the output of ps. With our applications that we're running, we can usually get away with roughly 5-10 context reloads. Do you have any 3rd party containers or such that live outside of the context (their JARs would be in CATALINA_HOME/server/lib or CATALINA_HOME/common/lib -- someone please confirm that the second of these two does *not* get discarded along with the rest of the context upon reload). If you are using 3rd party containers like this, they will not dump their contents when you re-load the servlet context. If you have static singleton instances of these, then you'll likely fill up memory all the time. This could also be related to the Java doesn't GC java.lang.Class objects thread that's going around this week, too. -chris - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
What apps are you running ? Maybe you are running something that consumes a lot of ram. Dhruva B. Reddy wrote: We get OutOfMemoryError's on Tomcat 4.1.27, running on RH 7.3, Sun JDK 1.4.1_02 (with 512MB allocated to it) The box has 1GB of RAM, 2GB of swap space, and four Xeon processors. This error seems to occur during periods of low load (like during the Thanksgiving holiday), after which Tomcat exits. According to the calls Runtime.freeMemory() and .totalMemory(), it comes nowhere near using that much (we track it every five minutes by hitting a servlet that calls these methods). Has anyone seen anything like this? Thanks, Dhruva __ Do you Yahoo!? Free Pop-Up Blocker - Get it now http://companion.yahoo.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
Dhruva B. Reddy wrote: We get OutOfMemoryError's on Tomcat 4.1.27, running on RH 7.3, Sun JDK 1.4.1_02 (with 512MB allocated to it) The box has 1GB of RAM, 2GB of swap space, and four Xeon processors. Are you using the context reloading feature? reloadable=true or using the tomcat manager app will cause this to happen eventually. This error seems to occur during periods of low load (like during the Thanksgiving holiday), after which Tomcat exits. According to the calls Runtime.freeMemory() and .totalMemory(), it comes nowhere near using that much (we track it every five minutes by hitting a servlet that calls these methods). Has anyone seen anything like this? Thanks, Dhruva __ Do you Yahoo!? Free Pop-Up Blocker - Get it now http://companion.yahoo.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ This communication is intended for the use of the recipient to whom it is addressed, and may contain confidential, personal, and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communications received in error, or subsequent reply, should be deleted or destroyed. --- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError
Have you updated the server settings? In our case the default ammount of RAM the VM was allowed to allocate was fixed to 64 mb (brrr)! Also watch out for the max. ammout of threads your server can create... this probably will be your next problem. Dhruva B. Reddy wrote: We get OutOfMemoryError's on Tomcat 4.1.27, running on RH 7.3, Sun JDK 1.4.1_02 (with 512MB allocated to it) The box has 1GB of RAM, 2GB of swap space, and four Xeon processors. This error seems to occur during periods of low load (like during the Thanksgiving holiday), after which Tomcat exits. According to the calls Runtime.freeMemory() and .totalMemory(), it comes nowhere near using that much (we track it every five minutes by hitting a servlet that calls these methods). Has anyone seen anything like this? Thanks, Dhruva __ Do you Yahoo!? Free Pop-Up Blocker - Get it now http://companion.yahoo.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError
Howdy, Does any one know of a neat way of monitoring memory usage, so that our servlet can decide to refuse a request if it senses that memory is about to run out? You have the two calls in the Runtime class (totalMemory() and freeMemory()), and that's pretty much it. This of course reflects only the heap. So you can have a background thread that checks freeMemory() every so often and if the free percentage falls below a certain configured threshold, sets some global flag variable that servlets (or better yet a simple filter) consults when deciding whether to allow servlet requests. free Memory = 172544, totalMemory = 6.6650112E7, maxMemory = 1.34217728E8 How were the above obtained? and I don't understand why the JVM didn't try to grab more memory when it could - is it because one of tomcat's threads unexpectedly ran out of memory that it should have been allocated? The JVM is constricted by how you configure it, e.g. the -Xmx parameter. If not specified, the JVM will allocate up to 64MB to the heap. thanks for any suggestions, This is a fragile situation with or without the above solution. Benchmark your system to know how many max concurrent requests you can support given your hardware, and make sure your customers/users know this number. You may also want to consider a script to monitor your log for OutOfMemoryError messages and restart tomcat or notify admins in this event. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError
Hi Yoav, You have the two calls in the Runtime class (totalMemory() and freeMemory()), and that's pretty much it. This of course reflects only the heap. So you can have a background thread that checks freeMemory() every so often and if the free percentage falls below a certain configured threshold, sets some global flag variable that servlets (or better yet a simple filter) consults when deciding whether to allow servlet requests. yup that was what I though of doing. I was surprised, though, that the JVM didn't try to increase the totalMemory for this app, though, as it looked like it could have. free Memory = 172544, totalMemory = 6.6650112E7, maxMemory = 1.34217728E8 How were the above obtained? I added a log4j message in the servlet code to report this whenever tuples are inserted by a user via our API. These were the last measurements before the OutOfMemoryError. Just before the crash, I could see freeMemory fluctuating around, and I guess at some point, it must have dipped below some threshold. I'm just now looking at the standard tomcat4 script in /etc/init.d to figure out whether any -Xmx parameters were set, but I guess the jvm thinks maxMemory is 134MB, looking at the numbers above. This is a fragile situation with or without the above solution. Benchmark your system to know how many max concurrent requests you can support given your hardware, and make sure your customers/users know this number. You may also want to consider a script to monitor your log for OutOfMemoryError messages and restart tomcat or notify admins in this event. good ideas, thanks for the suggestions. cheers, Andy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError under high load
Howdy, How much have you played around with the JVM tuning parameters? Yoav Shapira Millennium ChemInformatics -Original Message- From: Ian McFarland [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 11, 2003 8:05 AM To: Tomcat List Cc: Ian McFarland Subject: OutOfMemoryError under high load Hello, We're running into the following problem when our Tomcat instances are under heavy load. SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError: unable to create new native thread The exception is always on thread creation, and correlates a condition where the process count on the machine suddenly starts to ramp up from a few hundred to a little above 1000 processes. (Load is fairly constant during this time, and the condition always expresses itself at right around the same level of traffic, about 6Mbps of throughput per server.) There appears to be plenty of heap memory available when this happens, so that coupled with the thread creation message leads us to suspect that we're running into an OS tuning problem, instead of specifically a problem with our app. (A radical rewrite of how our domain objects are cached, and tuning of various caches to various sizes, resulted in near identical performance, aside from large differences in CPU utilization. This seems to further support this conclusion.) Tuning maxProcessors and a variety of other parameters in our app seems to have no discernible effect, neither beneficial nor detrimental, beyond changes in CPU utilization, which always falls in a band between 10 and 60%, depending on the tuning choices. We're currently using Tomcat 4.1.13 under Linux 2.4.17 with Sun JDK 1.4.1_01. We've tuned all the obvious parameters, (ulimit -n and /proc/sys/kernel/threads-max, for example,) or at least all the ones we could think of. I think we're missing something obvious, and the OS is not letting us spawn more threads. Anyone know what we're doing wrong? Thanks in advance, -Ian McFarland - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError under high load
Linux is the problem here. As you increase the mx JVM flag to be, the number of threads the java process can create decreases. This not a tomcat thing but a java on linux thing. For example, setting mx to be 1028 MBytes results in a thread ceiling of something like 427, which is really low for any loaded tomcat server. Unfortunately I don't have any solutions. You could switch to Unix, but that's hardly practical solution. I guess you could run two tomcat instances on the same server and load-balance somehow, but that sounds tough too. I never solved this one myself. -Original Message- From: Ian McFarland [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 11, 2003 5:05 AM To: Tomcat List Cc: Ian McFarland Subject: OutOfMemoryError under high load Hello, We're running into the following problem when our Tomcat instances are under heavy load. SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError: unable to create new native thread The exception is always on thread creation, and correlates a condition where the process count on the machine suddenly starts to ramp up from a few hundred to a little above 1000 processes. (Load is fairly constant during this time, and the condition always expresses itself at right around the same level of traffic, about 6Mbps of throughput per server.) There appears to be plenty of heap memory available when this happens, so that coupled with the thread creation message leads us to suspect that we're running into an OS tuning problem, instead of specifically a problem with our app. (A radical rewrite of how our domain objects are cached, and tuning of various caches to various sizes, resulted in near identical performance, aside from large differences in CPU utilization. This seems to further support this conclusion.) Tuning maxProcessors and a variety of other parameters in our app seems to have no discernible effect, neither beneficial nor detrimental, beyond changes in CPU utilization, which always falls in a band between 10 and 60%, depending on the tuning choices. We're currently using Tomcat 4.1.13 under Linux 2.4.17 with Sun JDK 1.4.1_01. We've tuned all the obvious parameters, (ulimit -n and /proc/sys/kernel/threads-max, for example,) or at least all the ones we could think of. I think we're missing something obvious, and the OS is not letting us spawn more threads. Anyone know what we're doing wrong? Thanks in advance, -Ian McFarland - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError under high load
Do you have any native code components? I had a serious problem with OutOfMemoryError: unable to create new native thread errors too. They pretty much went away when I ditched the Oracle OCI driver in favor of the thin driver. BTW, the default linux kernel puts a hard limit of ~1k threads on your processes. A simple java program that does nothing but spawn threads typically tops out around 970-980 threads on linux. From what our ops guys have said, this parameter cannot be changed without recompiling the kernel. Why on earth would you want more than 1000 threads? Surely you don't need that many Ajp/Coyote/whatever processors. Jeff Schnitzer [EMAIL PROTECTED] The Sims Online -Original Message- From: Ian McFarland [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 11, 2003 5:05 AM To: Tomcat List Cc: Ian McFarland Subject: OutOfMemoryError under high load Hello, We're running into the following problem when our Tomcat instances are under heavy load. SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError: unable to create new native thread The exception is always on thread creation, and correlates a condition where the process count on the machine suddenly starts to ramp up from a few hundred to a little above 1000 processes. (Load is fairly constant during this time, and the condition always expresses itself at right around the same level of traffic, about 6Mbps of throughput per server.) There appears to be plenty of heap memory available when this happens, so that coupled with the thread creation message leads us to suspect that we're running into an OS tuning problem, instead of specifically a problem with our app. (A radical rewrite of how our domain objects are cached, and tuning of various caches to various sizes, resulted in near identical performance, aside from large differences in CPU utilization. This seems to further support this conclusion.) Tuning maxProcessors and a variety of other parameters in our app seems to have no discernible effect, neither beneficial nor detrimental, beyond changes in CPU utilization, which always falls in a band between 10 and 60%, depending on the tuning choices. We're currently using Tomcat 4.1.13 under Linux 2.4.17 with Sun JDK 1.4.1_01. We've tuned all the obvious parameters, (ulimit -n and /proc/sys/kernel/threads-max, for example,) or at least all the ones we could think of. I think we're missing something obvious, and the OS is not letting us spawn more threads. Anyone know what we're doing wrong? Thanks in advance, -Ian McFarland - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError: cannot create new native thread
It sounds as if not the number of threads is the problem, but the amount of memory that is used by the threads. May be you should track the memory usage to verify that the memory is low when this Exception happens) (Just log the results of Runtime.[free|total]Memory()) -Original Message- From: Schnitzer, Jeff [mailto:[EMAIL PROTECTED]] Sent: Thursday, December 05, 2002 10:24 AM To: [EMAIL PROTECTED] Subject: OutOfMemoryError: cannot create new native thread What are the non-obvious causes of OutOfMemoryError: cannot create new native thread? This is happening quite a bit on my Tomcat cluster (linux + sun jdk1.4.1_01 + jboss/tomcat 4.0.4, no ejbs) under load, but the actual thread count (by either ps or the thread dump) is relatively low, often less than 200. Can anyone speculate on what might be wrong? I've checked, and just spawning threads my test went up to almost 1000 threads before complaining. It doesn't seem like I should run into problems at 200 threads. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: OutOfMemoryError: cannot create new native thread
OutOfMemoryError is a bad error description in java. It can mean any of the following: - Actually ran out of memory - Hit max # of threads allowed to run - Ran out of file handles - File handles include open files AND sockets - (?) Your JVM is bigger than the max process size allowed - Other things I don;t know There are tons of messages about dealing with all the above in the archives or google. -Tim Schnitzer, Jeff wrote: What are the non-obvious causes of OutOfMemoryError: cannot create new native thread? This is happening quite a bit on my Tomcat cluster (linux + sun jdk1.4.1_01 + jboss/tomcat 4.0.4, no ejbs) under load, but the actual thread count (by either ps or the thread dump) is relatively low, often less than 200. In the last thread dump, about half the Ajp13Processors were runnable in socketRead0(), and half were in wait mode like this: Ajp13Processor[8009][125] daemon prio=1 tid=0x0x8cd0b10 nid=0x374e in Object.wait() [7868..78680840] at java.lang.Object.wait(Native Method) - waiting on 0x47fdfb68 (a org.apache.ajp.tomcat4.Ajp13Processor) at java.lang.Object.wait(Object.java:426) at org.apache.ajp.tomcat4.Ajp13Processor.await(Ajp13Processor.java:316) - locked 0x47fdfb68 (a org.apache.ajp.tomcat4.Ajp13Processor) at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:542) at java.lang.Thread.run(Thread.java:536) Can anyone speculate on what might be wrong? I've checked, and just spawning threads my test went up to almost 1000 threads before complaining. It doesn't seem like I should run into problems at 200 threads. Thanks in advance, Jeff Schnitzer [EMAIL PROTECTED] The Sims Online -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: OutOfMemoryError after 88MB on Win2K / Tomcat 4.0.4
First of all, have you increased the memory available to Tomcat's JVM? You can do this by defining: JAVA_OPTS= -Xms128m -Xmx384m The numbers represent megabytes in that example. Xms is minimum and Xmx is maximum. Change the amounts to suit you. Another thing you might want to try is enabling Verbose Gargbage Collection. This will give you an ouput of what the garbage collector is doing. To turn it on, add this to the JAVA_OPTS: -verbose:gc Then, in catalina.out, you'll see status reports on the GC. Ian. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Thursday, October 03, 2002 5:37 PM To: [EMAIL PROTECTED] Subject: OutOfMemoryError after 88MB on Win2K / Tomcat 4.0.4 We are running a heavily XML-based application on Tomcat 4.0.4 on a dual P4 1.4 GHz box with 4GB of RAM. With 9 users hitting the application continuously, we experience OutOfMemoryErrors after tomcat.exe gets up to around 88MB of usage. Has anyone else experienced strange happenings with Tomcat on Win2K JDK 1.3.1 under load? We're wondering if the JVM is doing this, if there's something in Tomcat that is causing the memory to continuously creep up and never go down significantly, or if we're just really overlooking something in our application framework that is causing objects to not go out of scope. We haven't experienced this problem like this on other platforms (Linux and OS/400) and IBM is testing this application and not seeing the same kinds of results in WebSphere. I'd appreciate any feedback from other XML users here...we have a go-live date just a few days off and these problems are making my life very difficult. I'd like to at least be able to explain the situation, even if it is our fault. Not knowing is very frustrating. Thanks for your help! Jon Brisbin Sr. App. Developer EFCO Corporation www.efcocorp.com 417.235.3193 x2687
RE: OutOfMemoryError after 88MB on Win2K / Tomcat 4.0.4
Have you changed the default JVM options for Tomcat? Here are the default JVM options: -Xmsn Specify the initial size, in bytes, of the memory allocation pool. This value must be a multiple of 1024 greater than 1MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is 2MB. Examples: -Xms6291456 -Xms6144k -Xms6m -Xmxn Specify the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is 64MB. Examples: -Xmx83886080 -Xmx81920k -Xmx80m There are other JVM options available. They are documented here: http://java.sun.com/j2se/1.4.1/docs/tooldocs/windows/java.html For tomcat 4.x on linux, I made my changes in the setclasspath.sh file (I'm sure others have an opinion on this, but it works for me) by adding a statement at the top of the file: JAVA_OPTS=-verbose:gc -Xms64m -Xmx128m (The -verbose:gc prints a message to catalina.out whenever a gc occurs). hth, Mike Schulz -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Thursday, October 03, 2002 4:37 PM To: [EMAIL PROTECTED] Subject: OutOfMemoryError after 88MB on Win2K / Tomcat 4.0.4 We are running a heavily XML-based application on Tomcat 4.0.4 on a dual P4 1.4 GHz box with 4GB of RAM. With 9 users hitting the application continuously, we experience OutOfMemoryErrors after tomcat.exe gets up to around 88MB of usage. Has anyone else experienced strange happenings with Tomcat on Win2K JDK 1.3.1 under load? We're wondering if the JVM is doing this, if there's something in Tomcat that is causing the memory to continuously creep up and never go down significantly, or if we're just really overlooking something in our application framework that is causing objects to not go out of scope. We haven't experienced this problem like this on other platforms (Linux and OS/400) and IBM is testing this application and not seeing the same kinds of results in WebSphere. I'd appreciate any feedback from other XML users here...we have a go-live date just a few days off and these problems are making my life very difficult. I'd like to at least be able to explain the situation, even if it is our fault. Not knowing is very frustrating. Thanks for your help! Jon Brisbin Sr. App. Developer EFCO Corporation www.efcocorp.com 417.235.3193 x2687 -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: OutOfMemoryError after 88MB on Win2K / Tomcat 4.0.4
Yes, changing the heap was the first thing I changed and I actually thought of mentioning that because I figured that would be the immediate response :-) That wasn't the issue. We have discovered that this problem only manifests itself when using Tomcat as a service, with the provided tomcat.exe (and having JAVA_OPTS set in the environment to handle the heap size options). When started from the command line, with a minimum of 256mb on the heap, I get no problems with Tomcat past 150 MB of usage in Task Manager. We don't really like the idea of running it from a command prompt for maintenance purposes (we don't manage the box), but if some procedures are put in place, I don't see why it won't work in the short-term. I just didn't have time to figure out how to check and see if tomcat.exe was picking up my JAVA_OPTS environment variable and actually setting the heap size. If it wasn't, I would have thought I'd see problems after 64mb because I believe this is the default max on the heap size. BTW- The memory creep appears to be normal and will clear itself up after a long period of inactivity. I thought this was a leak on my part, but the GC wasn't doing anything until way later because the heap is so huge (I actually had to set a max of 1.5 GB when using catalina.bat run from DOS command line...the JVM gave me an error until this was set to some value...I eventually brought it down to this value, which was as high as I could get it without it giving me an error). Thanks for your help! Jon Brisbin Zabel, Ian [EMAIL PROTECTED] 10/03/2002 04:48 PM Please respond to Tomcat Users List To: 'Tomcat Users List' [EMAIL PROTECTED] cc: Fax to: Subject:RE: OutOfMemoryError after 88MB on Win2K / Tomcat 4.0.4 First of all, have you increased the memory available to Tomcat's JVM? You can do this by defining: JAVA_OPTS= -Xms128m -Xmx384m The numbers represent megabytes in that example. Xms is minimum and Xmx is maximum. Change the amounts to suit you. Another thing you might want to try is enabling Verbose Gargbage Collection. This will give you an ouput of what the garbage collector is doing. To turn it on, add this to the JAVA_OPTS: -verbose:gc Then, in catalina.out, you'll see status reports on the GC. Ian. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Thursday, October 03, 2002 5:37 PM To: [EMAIL PROTECTED] Subject: OutOfMemoryError after 88MB on Win2K / Tomcat 4.0.4 We are running a heavily XML-based application on Tomcat 4.0.4 on a dual P4 1.4 GHz box with 4GB of RAM. With 9 users hitting the application continuously, we experience OutOfMemoryErrors after tomcat.exe gets up to around 88MB of usage. Has anyone else experienced strange happenings with Tomcat on Win2K JDK 1.3.1 under load? We're wondering if the JVM is doing this, if there's something in Tomcat that is causing the memory to continuously creep up and never go down significantly, or if we're just really overlooking something in our application framework that is causing objects to not go out of scope. We haven't experienced this problem like this on other platforms (Linux and OS/400) and IBM is testing this application and not seeing the same kinds of results in WebSphere. I'd appreciate any feedback from other XML users here...we have a go-live date just a few days off and these problems are making my life very difficult. I'd like to at least be able to explain the situation, even if it is our fault. Not knowing is very frustrating. Thanks for your help! Jon Brisbin Sr. App. Developer EFCO Corporation www.efcocorp.com 417.235.3193 x2687
