I'm running a Tomcat/Apache web server and a PostgresQL database server. In the past two months or so, the number of webapps I'm running has drastically increased, and it has led to many unexpected problems. Please bear with me as a summarize before asking my question; hopefully this will prove useful to somebody someday:
machine 1: dual-933MHz, 512 MB RAM, Linux 2.4 (RedHat 7.1), Apache 1.3.19, Tomcat 4.0.3, Java 1.4 machine 2: 731MHz, 256 MB Ram, Linux 2.2 (RedHat 6.2), PostgresQL 7.1.3 The first problem I ran into was random crashes of PostgresQL. That apparently was caused by too many open files. I quadrupled the values in /proc/sys/fs/file-max and /proc/sys/fs/inode-max and have not had any problems with that machine since. The second problem turned up in my Tomcat logs: java.net.SocketException: Too many open files. Once I started seeing that, I had to reboot Tomcat. I upped /proc/sys/fs/file-max, but there is no /proc/sys/fs/inode-max, and that didn't seem to help. "ulimit -n 90000" seems to have fixed that, but... Tomcat kept crashing, but now it was throwing java.lang.OutOfMemoryError: requested x bytes. I added -Xms64M -Xmx256M to increase the Java heap size, and added some garbage collection routines. That didn't seem to fix it. Then I noticed this in one of the logs: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start at org.apache.catalina.connector.warp.WarpConnection.start (159) at org.apache.catalina.connector.warp.WarpConnector.run (601) at java.lang.Thread.run (536) Which prefigured an OutOfMemoryError and complete crash of Tomcat by minutes. Doing some research, I found that there was a built-in thread limit in glibc. I got the source RPM off the RedHat 7.1 CD, made the necessary changes in linuxthreads, and recompiled. I copied the new linuxthreads.so to /lib/i686, and everything seemed to be working fine. No crashes, but it was a weekend and nobody uses it on weekends. So this morning I did a 'make install' on glibc (huge mistake; I feel strongly that I had some reason for doing that but I can't remember what it was), and wham, Tomcat crashes and won't start up again. I reinstalled the glibc RPMs on the RedHat CD to try to get the system back to normal, but Tomcat still won't start up. Which is where my question comes in. Why won't it start up again? :) Using Java 1.4, Tomcat fails to start up with a segmentation fault. I switched to Java 1.3.1, and it fails to start up with a "cannot find libhpi.so". Java 1.2.2 works, but I'm using some 1.4 features, so that's only a temporary solution. I can still compile and run programs using 1.4; only Tomcat seems to not be working. What have I done and how can I fix it? Also, what more do I need to do to get these servers to be able to handle the load? Apache is serving about 400 requests per hour on average, as much as 1700 during a busy hour, which doesn't seem to me like it should be too much for this server to handle. I usually have between 80 and 120 concurrent database connections during peak hours, including sessions which are no longer in use and waiting to expire. Any help would be very much appreciated. Thanks, Ben Carterette [EMAIL PROTECTED] -- To unsubscribe: <mailto:[EMAIL PROTECTED]> For additional commands: <mailto:[EMAIL PROTECTED]> Troubles with the list: <mailto:[EMAIL PROTECTED]>