I found this interesting article about how Linux handles requests for memory, look at section "9.6 Overcommit and OOM": http://www.win.tue.nl/~aeb/linux/lk/lk-9.html I verified that our system runs with overcommit_memory = 0 and overcommit_ratio = 50. Which are the default values.
This post suggest to change these settings to 2 and 80 respectively, but we may not be able to start any new processes if we run out of memory (and therefore we may not be able to connect to the machine). http://www.hskupin.info/2010/06/17/how-to-fix-the-oom-killer-crashe-under-l inux/ Since we pre-allocate all the java heap memory (by setting -Xmx and -Xms to the same value), we accelerate the OOM killing the process. Therefore, the leak that is causing the problem just occur faster than if we only set the max value of the heap with -Xmx. Before I had made the recommendation to run with -Xmx and -Xms equal to the same value, but I think this works well in Solaris but not in Linux. Removing the -Xms option may give us just for more time between the occurrences of running out of memory. Nevertheless, I am finding that after removing the -Xms option, the process memory usage stabilizes and stops growing. -Jorge On Mon, Jun 11, 2012 at 11:01 AM, Jorge Medina <cerebrotecnolog...@gmail.com> wrote: > I'm finding it hard to believe, but all points that the problem was > the -Xms option of the Oracle (Sun) JVM. > I originally set it to the same value as -Xmx, so that all memory for > the heap is allocated when the JVM starts. > This works fine in Solaris, but it is not working in Ubuntu. > After removing that option, the JVM process memory usage seems to keep stable. > > I am using Sun JVM 1.6.0.26 > > -Jorge > > On Thu, Jun 7, 2012 at 11:38 PM, Pid * <p...@pidster.com> wrote: >> On 7 Jun 2012, at 23:03, Daniel Mikusa <dmik...@vmware.com> wrote: >> >>> ----- Original Message ----- >>>> Only 52 java threads. It used to fluctuate more (we made some >>>> changes >>>> to the app to perform a task in a single thread rather than spawning >>>> multiple threads, but the crash still occurs) . The number of threads >>>> is always below 100. >>>> >>>> jstack -F 21370 | grep ^Thread | wc -l >>>> ps -T -p 21370 (This gives me 63) >>>> >>>> I don't seem to specify the -Xss option: >>> >>> In some applications with a large number of threads (particularly when >>> running on 64-bit hardware) this setting can cause a problems. The default >>> value is pretty large (I think it's 1M on 64-bit systems). Since most apps >>> don't need that large of a value, an easy performance tuning step is to >>> lower the value of -Xss. >>> >>> Since you don't have very many threads, it seems unlikely that this is >>> causing your problem though. >>> >>> That being said, you could try explicitly setting a value for the thread >>> stack size. Finding the right values takes some testing though. I usually >>> start with something like 192k and run a few application tests. If I see >>> any stack overflow exceptions then I increase the value and rerun the >>> tests. Repeat until there are no stack overflow exceptions. >>> >>> >>> On a different note, what is the specific version of the JVM that you are >>> running? If it's not the latest, you could always try upgrading to the >>> latest version. >> >> You need to hook up the VisualVM + Memory Pools plugin. >> >> This will show you where the memory is being consumed, if it's by the JVM. >> >> >> p >> >> >> >>> >>> >>>> >>>> Xms6g -Xmx6g -XX:NewSize=4G -XX:MaxNewSize=4G -XX:SurvivorRatio=6 >>>> -XX:MaxPermSize=512M -XX:-UseConcMarkSweepGC -XX:+UseStringCache >>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/example/logs >>>> >>>> -Jorge >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jun 7, 2012 at 12:07 PM, Daniel Mikusa <dmik...@vmware.com> >>>> wrote: >>>>> ----- Original Message ----- >>>>>> I am using MongoDB through the Java driver allowing up to 100 >>>>>> connections to the MongoDB server. >>>>>> I also use DBCP with a max size of 50 JDBC connections. >>>>>> My webapp uses about 150 JAR files. >>>>>> There is no native libraries loaded from my webapp as far as I >>>>>> know. >>>>>> All the app is pure Java code. (Nevertheless, Tomcat is using the >>>>>> Tomcat Native Library) >>>>>> >>>>>> Is there a way I can monitor the number of file descriptors in use >>>>>> by >>>>>> the app? >>>>>> >>>>>> I have monitored the number of threads, but I haven't seen >>>>>> anything >>>>>> unusual. >>>>> >>>>> How many threads have you observed? Total threads, not just >>>>> threads for the connector. >>>>> >>>>> Also, what is the value you are using for thread stack size? -Xss >>>>> >>>>> Dan >>>>> >>>>> >>>>> >>>>> (but it could be that the burst is too fast to get catch by >>>>>> the monitoring tool) >>>>>> >>>>>> -Jorge >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 7, 2012 at 11:44 AM, Christopher Schultz >>>>>> <ch...@christopherschultz.net> wrote: >>>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>>> Hash: SHA1 >>>>>>> >>>>>>> Jorge, >>>>>>> >>>>>>> On 6/6/12 5:33 PM, Jorge Medina wrote: >>>>>>>> The web application uses Spring/Postgres/Mongo. >>>>>>> >>>>>>> Are you using MongoDB in-process or anything weird like that? Or >>>>>>> are >>>>>>> you connecting through some socket-based (or other) API? >>>>>>> >>>>>>>> It looks like a memory leak in native code, not java code; so >>>>>>>> my >>>>>>>> usual java toolset is not useful. >>>>>>> >>>>>>> If what you are observing is accurate (non-heap memory grows, >>>>>>> heap >>>>>>> stays reasonable) then it will definitely be more difficult to >>>>>>> track-down. >>>>>>> >>>>>>>> Tomcat runs behind nginx in a EC2 instance. The application >>>>>>>> uses >>>>>>>> Sun (now Oracle) JDK 1.6. >>>>>>>> >>>>>>>> Any suggestions on what should I look at? >>>>>>> >>>>>>> What do your <Connectors> look like? How many JDBC connections >>>>>>> do >>>>>>> you >>>>>>> have in your connection pool (which you are hopefully using!)? >>>>>>> How >>>>>>> about the same equivalent for MongoDB? >>>>>>> >>>>>>> Does your webapp keep lots of files open? Do you have an >>>>>>> unusually-large number of JAR files in your webapp? Do you have >>>>>>> any >>>>>>> native libraries in use within your webapp? >>>>>>> >>>>>>> What are all the non-default system properties that you are >>>>>>> setting >>>>>>> at >>>>>>> JVM launch time (you can easily see this from a 'ps' list)? >>>>>>> >>>>>>> Two things that can eat-up native memory fast in a JVM are file >>>>>>> descriptors and threads, so let's start there. >>>>>>> >>>>>>> - -chris >>>>>>> -----BEGIN PGP SIGNATURE----- >>>>>>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin) >>>>>>> Comment: GPGTools - http://gpgtools.org >>>>>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ >>>>>>> >>>>>>> iEYEARECAAYFAk/Q9ooACgkQ9CaO5/Lv0PDPyQCfVtddxMDOgQbjmMGC3gvnK+Qq >>>>>>> aZMAnjVu67+9Sm2bdYzAd91ZOrYo3DFI >>>>>>> =r+vl >>>>>>> -----END PGP SIGNATURE----- >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
