Re: Preventing OutOfMemoryError: Java heap space
Dear Todd, I'm not sure I can use session counting, as my session size is not consistent. I could try to estimate the size of each session, and keep a global counter, but that seems like a lot of work. JMX offers quite a bit of insight in what your GC is doing. You could implement throttling as a function of the number of full GC's that your system experiences. While full GC's do happen from time to time, you could trigger if they happen more than once per so-many seconds. Actual values would probably be highly application-specific. I agree with someone else that 256MB is not a whole lot or RAM for a busy server. With current prices of RAM I would go for a gig or two and take it from there. I understand that garbage collection is, err, whimsical, but I think I'm going to give it a go anyway. I will keep in mind that the memory results from Runtime will probably under-report the available memory. I'm going to add an explicit request for garbage collection, _before_ the memory becomes seriously depleted. Here is what happens if you do http://java-monitor.com/forum/showthread.php?t=188 It is not pretty, I can tell you that. I add -XX:-DisableExplicitGC to my JAVA_OPTS for this very reason. -- Kees Jan http://java-monitor.com/forum/ kjkos...@kjkoster.org 06-51838192 The secret of success lies in the stability of the goal. -- Benjamin Disraeli - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
> From: André Warnier [mailto:a...@ice-sa.com] > Thanks, Chuck and Peter, for the clarifications on OOM. > I believe that unconsciously, with my "large object > reservation" theory, > I was vaguely remembering something I had read some time in the past. > So I searched Google for "java +parachute +memory" and this > is something > I found : > > http://mail-archives.apache.org/mod_mbox/tomcat-dev/200703.mbo x/%3c20070325171940.34dae1a9...@eris.apache.org%3e > > Does this have any bearing on the OP's issue ? > > It is past me capacities to see if it is relevant or not (I don't mean > the above post per se, but the idea). Nice. The idea's relevant... as long as you realise that all it's doing is defending a very few critical areas of Tomcat's code against some (not all) OOMEs. Another thread could allocate heap memory between lines in this code, leading to unexpected failures because the memory freed by the parachute has been used elsewhere; and it doesn't defend all areas of Tomcat's code (I think - Filip will no doubt correct me). It's a great way of helping to make sure that, most of the time, it's at least possible to log an OOME - assuming the application passes it up the stack, which most will. It's not a general solution without a lot more work; each webapp would have to do something similar at each point that it might allocate heap memory. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Thanks, Chuck and Peter, for the clarifications on OOM. I believe that unconsciously, with my "large object reservation" theory, I was vaguely remembering something I had read some time in the past. So I searched Google for "java +parachute +memory" and this is something I found : http://mail-archives.apache.org/mod_mbox/tomcat-dev/200703.mbox/%3c20070325171940.34dae1a9...@eris.apache.org%3e Does this have any bearing on the OP's issue ? It is past me capacities to see if it is relevant or not (I don't mean the above post per se, but the idea). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
> From: Todd Hivnor [mailto:spambox_98...@yahoo.com] > Subject: Re: Preventing OutOfMemoryError: Java heap space > > In my tests, as the RAM becomes depleted, the server > response becomes slower and slower. That's because you're going through garbage collections at an ever increasing frequency, with little benefit. > Yes of course I can throw more RAM at the problem. > But I would still like to have a graceful response to the > "overloaded" scenario. More RAM is just delaying the > problem. You need to size the system for the expected load, and refuse requests when you approach some arbitrary threshold. You also need to check very carefully for memory leaks in your webapps, since your symptoms are typical of that kind of problem. If you aren't doing so already, run in a 64-bit environment, allowing a much larger heap; your current setting of -Xmx256m is tiny in today's world. Just make sure you have enough RAM to support the heap, otherwise paging will kill your performance. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
On 13.05.2009 08:07, Peter Crowther wrote: >> From: André Warnier [...@ice-sa.com] would it not be easier to catch >> the OOM exception and then return a "sorry, server overloaded" page >> to the browser ? > > At that point, it's too late. A thread, somewhere in the system, > tried to allocate some memory for an object and couldn't. This could > happen anywhere! It might be while allocating the buffer to read the > request, or the buffer to send the response, or even the socket > instance from which to read the request. > > There are then no guarantees. You may be unable to send anything > back to the client, as you may be unable to access the socket because > the OOM was caused by trying to create that socket. Or the thread > that got the exception might be one of the threads processing > incoming requests. Once the system encounters an out-of-memory > error, you pretty much have to stop the process and start again, as > you generally have no idea what else has failed. You can't even save > any outstanding work to file, as there's no guarantee you can > allocate the memory for the file object - if MS Word runs out of > memory, you just lost your document! > > This isn't unique to Java, by the way - if you want a coding > nightmare, try to handle *all* malloc() fails in a reasonably sized C > program. Or kalloc() calls in a kernel :-). When you realise that > running out of memory means you can no longer guarantee to allocate > any new memory for any purpose at all, the scale of the problem > becomes apparent. +1! Another example for "no more guarantees": after an OOME in heap, your business logic might be broken in a very unexpected way! Don't let any Java process do more work once it had an OOME in heap. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
> From: Todd Hivnor [mailto:spambox_98...@yahoo.com] > One challenge with Peter's suggestion of tracking the > number of sessions myself is that I have a collection > of webapps. So I can't just set up a shared static counter; > I need a counter which works across multiple webapps. > The only way I know to do that is to use a text file, > and take care about locking the file before updating it. > Or perhaps I can use ServletContext. Some classes can be shared across webapps, though the precise approach varies depending on the version. See the "Classloader how-to" for your Tomcat version at http://tomcat.apache.org for details. Pre-6.0, you could load a class through the common or shared classloaders; in 6.0, your options are reduced somewhat. That class could keep a common integer. It doesn't have to be a straight session count; you could weight it based on the relative sizes of the sessions. A heavy session increments the counter by 10, a light one by 1. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
> From: André Warnier [...@ice-sa.com] > would it not be easier to catch the OOM exception and then return a > "sorry, server overloaded" page to the browser ? At that point, it's too late. A thread, somewhere in the system, tried to allocate some memory for an object and couldn't. This could happen anywhere! It might be while allocating the buffer to read the request, or the buffer to send the response, or even the socket instance from which to read the request. There are then no guarantees. You may be unable to send anything back to the client, as you may be unable to access the socket because the OOM was caused by trying to create that socket. Or the thread that got the exception might be one of the threads processing incoming requests. Once the system encounters an out-of-memory error, you pretty much have to stop the process and start again, as you generally have no idea what else has failed. You can't even save any outstanding work to file, as there's no guarantee you can allocate the memory for the file object - if MS Word runs out of memory, you just lost your document! This isn't unique to Java, by the way - if you want a coding nightmare, try to handle *all* malloc() fails in a reasonably sized C program. Or kalloc() calls in a kernel :-). When you realise that running out of memory means you can no longer guarantee to allocate any new memory for any purpose at all, the scale of the problem becomes apparent. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
>> But would it not be easier to catch the OOM exception and then >> return a "sorry, server overloaded" page to the browser ? > It's difficult to do that when the OOME may occur in Tomcat code, > outside of control of the webapp. Wow I had assumed I could always catch this type of exception. Thanks for clarifying. In any case, tho, I specifically don't want the OOME exception to occur, largely because it takes so much _time_ to occur. In my tests, as the RAM becomes depleted, the server response becomes slower and slower. The actual OOME takes over a minute to appear. Apparently the JVM is making a heroic effort to satisfy the request. >> (I would mischievously also add a suggestion to buy some more RAM, >> as an even cheaper alternative). >That's the only reasonable solution, if the OP is unwilling to > implement the throttling Peter suggested. Yes of course I can throw more RAM at the problem. But I would still like to have a graceful response to the "overloaded" scenario. More RAM is just delaying the problem. I was hoping there was a simple solution I had overlooked. One challenge with Peter's suggestion of tracking the number of sessions myself is that I have a collection of webapps. So I can't just set up a shared static counter; I need a counter which works across multiple webapps. The only way I know to do that is to use a text file, and take care about locking the file before updating it. Or perhaps I can use ServletContext. Overall session counting sounds like my best option. Again, thanks for all the clues & suggestions. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
> From: André Warnier [mailto:a...@ice-sa.com] > Subject: Re: Preventing OutOfMemoryError: Java heap space > > > I'm going to add an explicit request for garbage > > collection, _before_ the memory becomes seriously > > depleted. To the OP: explicitly calling GC is a complete waste of time. You have to do something to free up memory by shutting down sessions or otherwise terminating activity within the JVM. Your calling GC won't accomplish anything that the JVM automatically calling GC doesn't do. > But would it not be easier to catch the OOM exception and then > return a "sorry, server overloaded" page to the browser ? It's difficult to do that when the OOME may occur in Tomcat code, outside of control of the webapp. > (and maybe then also trigger a GC just for tidiness) Again, a waste of time - the OOME is not presented until after a major GC has been attempted. > Then this thread starts listening for OOM exceptions (I just > imagine that this can be done). I don't know of any means to do that. > (I would mischievously also add a suggestion to buy some more RAM, > as an even cheaper alternative). That's the only reasonable solution, if the OP is unwilling to implement the throttling Peter suggested. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Todd Hivnor wrote: From: Todd Hivnor [spambox_98...@yahoo.com] I would like to proactively avoid running out of heap space. I would like people get a "Server Too Busy" message, _before_ the heap is actually exhausted. I would rather serve 40 users well than 45 users poorly. Rather than monitor memory, which is subject to GC whims as George points out, I'd suggest monitoring the number of sessions in your application - find a number of sessions you can live with, keep a (suitably synchronized) counter that's incremented and decremented as sessions are created and destroyed, and don't start new sessions if that counter is higher than your threshold. - Peter Peter / George, thanks for the input. I'm not sure I can use session counting, as my session size is not consistent. I could try to estimate the size of each session, and keep a global counter, but that seems like a lot of work. I understand that garbage collection is, err, whimsical, but I think I'm going to give it a go anyway. I will keep in mind that the memory results from Runtime will probably under-report the available memory. I'm going to add an explicit request for garbage collection, _before_ the memory becomes seriously depleted. Thanks for the suggestions! Sorry for jumping in as a naive and frankly incompetent java and Tomcat programmer, and do not feel obliged to respond if the question is really stupid. But would it not be easier to catch the OOM exception and then return a "sorry, server overloaded" page to the browser ? (and maybe then also trigger a GC just for tidiness) Or a more ambitious scheme : at Tomcat startup, a thread is created which immediately allocates some largish object and keeps it. Then this thread starts listening for OOM exceptions (I just imagine that this can be done). When one occurs, the thread releases the object and triggers a GC. Then it tries to allocate the object again. Then.. I'm running out of ideas and technique. It was just a thought, in the direction of making GC a bit more deterministic. (I would mischievously also add a suggestion to buy some more RAM, as an even cheaper alternative). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
>> From: Todd Hivnor [spambox_98...@yahoo.com] >> I would like to proactively avoid running out of heap >> space. I would like people get a "Server Too Busy" >> message, _before_ the heap is actually exhausted. > >> I would rather serve 40 users well than 45 users >> poorly. > Rather than monitor memory, which is subject to GC > whims as George points out, I'd suggest monitoring > the number of sessions in your application - find a > number of sessions you can live with, keep a (suitably > synchronized) counter that's incremented and decremented > as sessions are created and destroyed, and don't start > new sessions if that counter is higher than your threshold. > - Peter Peter / George, thanks for the input. I'm not sure I can use session counting, as my session size is not consistent. I could try to estimate the size of each session, and keep a global counter, but that seems like a lot of work. I understand that garbage collection is, err, whimsical, but I think I'm going to give it a go anyway. I will keep in mind that the memory results from Runtime will probably under-report the available memory. I'm going to add an explicit request for garbage collection, _before_ the memory becomes seriously depleted. Thanks for the suggestions! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
> From: Todd Hivnor [spambox_98...@yahoo.com] > I would like to proactively avoid running out of heap > space. I would like people get a "Server Too Busy" > message, _before_ the heap is actually exhausted. > I would rather serve 40 users well than 45 users > poorly. Rather than monitor memory, which is subject to GC whims as George points out, I'd suggest monitoring the number of sessions in your application - find a number of sessions you can live with, keep a (suitably synchronized) counter that's incremented and decremented as sessions are created and destroyed, and don't start new sessions if that counter is higher than your threshold. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
I think the results are going to be pretty erratic. The issue that I see is that the garbage collector operation is (to my knowledge) not deterministic. IOW, you're not really accounting for memory that could be garbage collected. So, I think that you'll have a systematic bias showing less memory than is actually available. I suppose you could throw in Runtime.getRuntime().runFinalization() Runtime.getRuntime().gc() However, runtime.gc() just "suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse" Todd Hivnor wrote: I have a Java application running under Tomcat 6.0.18 on Ubuntu. This is using Sun's 1.6.0_07 JVM. I know how to set the max heap space by setting -Xmx256m in CATALINA_OPTS. But with a lot of sessions, I still have the possibility of running out of heap space. My application uses a lot of memory per user. I know I should avoid this, but at this point I can't dramatically change the software. And most of the time we have a small number of users anyway. I would like to proactively avoid running out of heap space. I would like people get a "Server Too Busy" message, _before_ the heap is actually exhausted. I would rather serve 40 users well than 45 users poorly. I'm wondering if there are any Tomcat configuration options, or filters, or whatever, which might address this. My current idea, which I think is solid, is to check the available memory before populating any new Session instances. My current test uses the following formula to determine the amount of space available: Runtime.getRuntime().maxMemory() -Runtime.getRuntime().totalMemory() +Runtime.getRuntime().freeMemory(); In other words, take the amount of memory we COULD allocate, subtract the amount we HAVE allocated, and then add back the amount that is allocated but unused. To phrase it another way, add the amount of free memory to the amount of unallocated memory. In practice this formula approaches zero as I get closer and closer to running out of heap space. So it seems to be correct. I’m wondering what people think of this idea, or if there is a better / simpler way to accomplish my goal. Opinions? Ideas? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- George Sexton MH Software, Inc. Voice: +1 303 438 9585 URL: http://www.mhsoftware.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Preventing OutOfMemoryError: Java heap space
I have a Java application running under Tomcat 6.0.18 on Ubuntu. This is using Sun's 1.6.0_07 JVM. I know how to set the max heap space by setting -Xmx256m in CATALINA_OPTS. But with a lot of sessions, I still have the possibility of running out of heap space. My application uses a lot of memory per user. I know I should avoid this, but at this point I can't dramatically change the software. And most of the time we have a small number of users anyway. I would like to proactively avoid running out of heap space. I would like people get a "Server Too Busy" message, _before_ the heap is actually exhausted. I would rather serve 40 users well than 45 users poorly. I'm wondering if there are any Tomcat configuration options, or filters, or whatever, which might address this. My current idea, which I think is solid, is to check the available memory before populating any new Session instances. My current test uses the following formula to determine the amount of space available: Runtime.getRuntime().maxMemory() -Runtime.getRuntime().totalMemory() +Runtime.getRuntime().freeMemory(); In other words, take the amount of memory we COULD allocate, subtract the amount we HAVE allocated, and then add back the amount that is allocated but unused. To phrase it another way, add the amount of free memory to the amount of unallocated memory. In practice this formula approaches zero as I get closer and closer to running out of heap space. So it seems to be correct. I’m wondering what people think of this idea, or if there is a better / simpler way to accomplish my goal. Opinions? Ideas? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
