-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Bill,
On 12/9/13, 8:20 PM, Bill Davidson wrote: > On 12/9/2013 3:12 PM, Christopher Schultz wrote: >> >> Was it a transient error, or a chronic condition? A single thread >> can, for instance, spew objects into its stack or eden space >> exhausting memory but, when that thread hits the OOME, all those >> objects are freed which basically recovers from the situation. >> >> If, instead, you fill-up some shared cache, buffer, etc. and NO >> threads can get more memory, then you're basically toast. >> >> Which of the above was it? > > It looked more like the first one though we still haven't tracked > down the cause. We had several dozen threads running at the time. > That's common for us. It's not that unusual for us to have a > couple of hundred users with active sessions per server at any > given time. > >> There are a bunch of things you can try to do. They all have >> their caveats, failure scenarios, and inefficacies. >> >> 1. Use -XX:OnOutOfMemoryError="cmd args;cmd args" >> >> Rig this to email you, register a passive-check data point with >> your monitoring server, etc. Just remember that OOMEs happen for >> a number of reasons. You could have run out of file handles or >> you could have run out of heap space. > > That looks interesting. It wouldn't tell me about the error but > at least I'd know that there was an OOME. Better than nothing and > I can go check catalina.out. Of course, I still have the problem > that threads silently fail and show my users not so much as an > error message. > >> 2. Use JMX monitoring, set java.lang:MemoryPool/[heap >> space]/UsageThreshold to whatever byte value you want to set as >> your limit. Then, check java.lang:MemoryPool/[heap >> space]/UsageThresholdExceeded to see if it is true. If so, your >> usage threshold has been exceeded. >> >> Note that this is not proof-positive than an OOME occurred. It's >> also tough to tell what value to use for the threshold. You can't >> really set it to MaxHeap - 1 byte, because you'll never get that >> value in practice. If you set it too low, you'll get warnings all >> the time when your heap usage rises in the normal course of >> business. > > I'm less enthused about that one. > >> 3. catch IOException in a filter and set an application >> attribute. Check this attribute from your monitor. >> >> I've been considering doing this, because I can rig it so that >> the error handler does not actually require any memory to run. >> The problem is that sometimes OOMEs interrupt one thread and not >> another. You may not catch the OOME in that thread -- it may >> happen in a background thread that does not go through the >> filter. > > I'm not sure I understand this one. How does an IOException relate > to an OOME? Sorry, I meant of course OutOfMemoryError. Just make sure you use as little memory as possible during the exception handler or it will fail itself and possibly mask the original problem. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJSqIFyAAoJEBzwKT+lPKRYKxoP/iDp+6bzkKN3KgqsMziO2fSl /NTbWIz1F9jzBAkFpRs+MQqwT3ZzUbE0QPbTnxPh3ShS+iD+UobtvdS2mn6YLNX9 qcWislmZKYMSFK/idc9JgBZE6XDRPG6bAo/X2lEfV9rURHVaA10QCgt7xDXdfN9b 5Ggs0ZfA5v2VQNIZyDFIukZzswFfA/VVb42vAR/wkTuBVVT/opZliv19gHc6y9D9 harx4z5cfEFQMq8YLdrQJIyPUXeoRoS9Um9oujS2FCWDEa0kni5Fn2nFh7beHHzi CMjncYAQ57pm59LwWp0PzRsZmPsr/UmgwiM95yt+c6cqxZKCr3d9xqqAMaqC33Fj sK8Hz/JzFkkfAK5xRHmdSqP+svpFpAvWJD75LOr4XvEOmEM1hCe4hkbzsa6wDJ5y f7EHAVbCjbgqtE7Ic3G5Gxz7KlarlWn7QIvJHJfy1kD9KbSVJtph5O6d6fQkxaQG vxeL9LqnMmPq8YmAlvuK/V2uUAmHNS4TVYKYTLnx/Z9kuiozhQSGoJAWP36KpM+3 kXZi9/9/L5SIry+R+PABj8UIKktIZ9ZJGZzV9uT8SiiEdkADz8kgcRWRUtdk3eDD T9Dzt2Tr9SV+HKUVkE3MTpqpAO6NVroOZ57ij6diiD7WXEMb9WeZcUt7IxxNOf+A nDT5bAeuQLHwAZTyQN74 =N4NL -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
