----- Original Message ----- > Only 52 java threads. It used to fluctuate more (we made some > changes > to the app to perform a task in a single thread rather than spawning > multiple threads, but the crash still occurs) . The number of threads > is always below 100. > > jstack -F 21370 | grep ^Thread | wc -l > ps -T -p 21370 (This gives me 63) > > I don't seem to specify the -Xss option:
In some applications with a large number of threads (particularly when running on 64-bit hardware) this setting can cause a problems. The default value is pretty large (I think it's 1M on 64-bit systems). Since most apps don't need that large of a value, an easy performance tuning step is to lower the value of -Xss. Since you don't have very many threads, it seems unlikely that this is causing your problem though. That being said, you could try explicitly setting a value for the thread stack size. Finding the right values takes some testing though. I usually start with something like 192k and run a few application tests. If I see any stack overflow exceptions then I increase the value and rerun the tests. Repeat until there are no stack overflow exceptions. On a different note, what is the specific version of the JVM that you are running? If it's not the latest, you could always try upgrading to the latest version. Dan > > Xms6g -Xmx6g -XX:NewSize=4G -XX:MaxNewSize=4G -XX:SurvivorRatio=6 > -XX:MaxPermSize=512M -XX:-UseConcMarkSweepGC -XX:+UseStringCache > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/example/logs > > -Jorge > > > > > > > On Thu, Jun 7, 2012 at 12:07 PM, Daniel Mikusa <dmik...@vmware.com> > wrote: > > ----- Original Message ----- > >> I am using MongoDB through the Java driver allowing up to 100 > >> connections to the MongoDB server. > >> I also use DBCP with a max size of 50 JDBC connections. > >> My webapp uses about 150 JAR files. > >> There is no native libraries loaded from my webapp as far as I > >> know. > >> All the app is pure Java code. (Nevertheless, Tomcat is using the > >> Tomcat Native Library) > >> > >> Is there a way I can monitor the number of file descriptors in use > >> by > >> the app? > >> > >> I have monitored the number of threads, but I haven't seen > >> anything > >> unusual. > > > > How many threads have you observed? Total threads, not just > > threads for the connector. > > > > Also, what is the value you are using for thread stack size? -Xss > > > > Dan > > > > > > > > (but it could be that the burst is too fast to get catch by > >> the monitoring tool) > >> > >> -Jorge > >> > >> > >> > >> > >> > >> > >> > >> On Thu, Jun 7, 2012 at 11:44 AM, Christopher Schultz > >> <ch...@christopherschultz.net> wrote: > >> > -----BEGIN PGP SIGNED MESSAGE----- > >> > Hash: SHA1 > >> > > >> > Jorge, > >> > > >> > On 6/6/12 5:33 PM, Jorge Medina wrote: > >> >> The web application uses Spring/Postgres/Mongo. > >> > > >> > Are you using MongoDB in-process or anything weird like that? Or > >> > are > >> > you connecting through some socket-based (or other) API? > >> > > >> >> It looks like a memory leak in native code, not java code; so > >> >> my > >> >> usual java toolset is not useful. > >> > > >> > If what you are observing is accurate (non-heap memory grows, > >> > heap > >> > stays reasonable) then it will definitely be more difficult to > >> > track-down. > >> > > >> >> Tomcat runs behind nginx in a EC2 instance. The application > >> >> uses > >> >> Sun (now Oracle) JDK 1.6. > >> >> > >> >> Any suggestions on what should I look at? > >> > > >> > What do your <Connectors> look like? How many JDBC connections > >> > do > >> > you > >> > have in your connection pool (which you are hopefully using!)? > >> > How > >> > about the same equivalent for MongoDB? > >> > > >> > Does your webapp keep lots of files open? Do you have an > >> > unusually-large number of JAR files in your webapp? Do you have > >> > any > >> > native libraries in use within your webapp? > >> > > >> > What are all the non-default system properties that you are > >> > setting > >> > at > >> > JVM launch time (you can easily see this from a 'ps' list)? > >> > > >> > Two things that can eat-up native memory fast in a JVM are file > >> > descriptors and threads, so let's start there. > >> > > >> > - -chris > >> > -----BEGIN PGP SIGNATURE----- > >> > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > >> > Comment: GPGTools - http://gpgtools.org > >> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > >> > > >> > iEYEARECAAYFAk/Q9ooACgkQ9CaO5/Lv0PDPyQCfVtddxMDOgQbjmMGC3gvnK+Qq > >> > aZMAnjVu67+9Sm2bdYzAd91ZOrYo3DFI > >> > =r+vl > >> > -----END PGP SIGNATURE----- > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> > For additional commands, e-mail: users-h...@tomcat.apache.org > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
