Background:
We have a moderately high traffic web application (between 8 to 21 million
hits/day) running Apache to serve static content (also to load balance and
create a DMZ) and Weblogic to serve dynamic content (Struts 1.1 based Java web
application).
We are trying to replace Weblogic with Tomcat and we have ported our code to
work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING
environment as long as there is no real load.
The Issue - Load Testing:
In our staging environment for load testing, when we run the load test using
525 concurrent users, the app doesn't perform at all. The CPU usage (on Apache
and Tomcat Servers) hovers between 7% to 8%. The database server CPU usage is
also between 4 and 5%.
Setup for Load Testing:
We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), 2
Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). Each
server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and Apache.
Mentioned below is the version information:
Apache Version 2.2.14 (with mod_jk module)
Tomcat: 6.0.29
Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bits
Connection Pool: DBCP.
Mentioned below are connector settings in conf/server.xml:
<Connector
address="stagingTCserver01"
backlog="300"
connectionTimeout="60000"
enableLookups="false"
maxPostSize="2097152"
maxSpareThreads="10"
maxThreads="30"
minSpareThreads="5"
port="8006"
protocol="AJP/1.3"
tcpNoDelay="true"
/>
Mentioned below are the settings for JNDI resource configured in
conf/context.xml:
<Resource
name="jdbc/onlinedb"
auth="Container"
type="javax.sql.DataSource"
driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB"
initialSize="1"
maxActive="30"
minIdle="1"
maxIdle="5"
maxWait="300000"
poolPreparedStatements="true"
maxOpenPreparedStatements="300"
validationQuery="SELECT 1 FROM BB_DUAL"
testOnBorrow="true"
validationInterval="10000"
testWhileIdle="true"
/>
JVM Parameters:
-Xms512m -Xmx2048m
-XX:PermSize=128m -XX:MaxPermSize=256m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/
-verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log
It must be noted that Weblogic setup performs very well using similar settings.
Garbage Collection: While the load test is running, Garbage collection works
just fine i.e. Young GC occurring every 2-3 minutes and takes less than half a
second. Full GC occurs every hour and takes a little over 2 seconds.
Any tips/pointers will be greatly appreciated.
Talha.
