Hello Talha, On Fri, Sep 2, 2011 at 11:34 PM, Talha Fazal <tfa...@credera.com> wrote: > Q. Have you tried running directly against tomcat without apache inbetween? > A. Yes; this resolves the performance bottleneck.
Than its pretty clear where to search, isn't it? > > Q. <holywar>from my experience there is no need for apache in your setup > anyway </holywar>. > A. :-) The reasons for having apache in front: >A) Create a DMZ and protect Tomcat from being exposed to outside attacks; You know that httpd does nothing but forwards each and single request to tomcat? So you are not really protect them, you just add another (time consuming) layer. > B) Load balancing. You mentioned that you have 2 httpd servers, so all the traffic to 6 (or was it eight) tomcat servers are going through the two httpds? And how do you loadbalancer the 2 httpd servers? Regards Leon > > Thanks Leon! > > -----Original Message----- > From: Leon Rosenberg [mailto:rosenberg.l...@gmail.com] > Sent: Friday, September 02, 2011 3:19 PM > To: Tomcat Users List > Subject: Re: Tomcat Performance Turning. > > Hello Talha, > > On Fri, Sep 2, 2011 at 6:16 PM, Talha Fazal <tfa...@credera.com> wrote: >> PLEASE SEE MY ANSWERS BELOW IN UPPER CAPS. >> >> ONE IMPORTANT OBSERVATION: Even though, tomcat's response time is decent, >> for some reason, apache is taking a lot longer to serve requests. We are >> using mod_jk with Tomcat using AJP1.3 protocol in the connector setting in >> server.xml. We plan to try mod_proxy_ajp and mod_proxy instead of mod_jk. >> Any thoughts here? > > Interesting observation. Have you tried running directly against > tomcat without apache inbetween? > <holywar>from my experience there is no need for apache in your setup > anyway </holywar>. > > In the past we used both mod_jk and mod_proxy_ajp, and even > mod_proxy_ajp is somewhat faster, the difference where never 17 > seconds. > Your best guess would probably be to test without httpd to rule out if > tomcat or httpd is the bottleneck. > > regards > Leon > >> >> Thanks, >> >> >> -----Original Message----- >> From: R Batchelor [mailto:rsbat...@yahoo.com] >> Sent: Friday, September 02, 2011 10:10 AM >> To: Tomcat Users List >> Subject: Re: Tomcat Performance Turning. >> >> Have you ruled out issues with db connection pooling? NO. IN FACT, THERE ARE >> KNOWN DBCP CONNECTION POOLING DEADLOCK ISSUES IN TOMCAT 6.0.26 >> [https://issues.apache.org/jira/browse/DBCP-270]. I AM LOBBYING TO UPGRADE >> TO TOMCAT 7.0.20 AND USE THE JDBC CONNECTION POOL INSTEAD OF DBCP CONNECTION >> POOL. >> >> You might consider setting your dbcp maxWait to 8000 or less and watch >> for "timeout waiting for idle object" exceptions. Also, you could >> monitor database connections/active-users on the DB side to see if your >> dbcp pools are max'd but all the database users/connections are idle. >> These can be symptoms of an application mis-managing its dbcp pools. >> I WILL DEFINITELY TRY THIS OUT. THANKS! >> >> ________________________________ >> From: Leon Rosenberg <rosenberg.l...@gmail.com> >> To: Tomcat Users List <users@tomcat.apache.org> >> Sent: Friday, September 2, 2011 7:55 AM >> Subject: Re: Tomcat Performance Turning. >> >> Hello Talha, >> >> well... the quickshots are through. >> You should really create some threaddumps after each other (for >> example with jstack) and try to find out which thread is slowing the >> app down. >> For starters you could try with code you changed for tomcat >> adaptations if any. The problem seems to lie beneath the surface, so >> you will have to start >> performance monitoring and look into the inside. >> However, time difference of 17 seconds must be something very very >> obvious like a synchronized block in a valve/filter, db lock (have you >> checked the db locks?), or an if (tomcat) >> Thread.sleep(1000L*60*17);.... >> >> regards >> Leon >> >> On Fri, Sep 2, 2011 at 4:21 PM, Talha Fazal <tfa...@credera.com> wrote: >>> Hi Leon, >>> >>> Please see any answers in CAPS below. >>> >>> >>> >>> -----Original Message----- >>> From: Leon Rosenberg [mailto:rosenberg.l...@gmail.com] >>> Sent: Friday, September 02, 2011 9:06 AM >>> To: Tomcat Users List >>> Subject: Re: Tomcat Performance Turning. >>> >>> Hello Talha, >>> >>> seems that your tomcats are fine. >>> Question, you mention that the tomcat do not perform, but you don't >>> tell us how you come to this conclusion except for cpu load. ANSWER: >>> LOOKING INTO THE TOMCAT ACCESS LOGS, THE RESPONSE TIME (%D) IS HORRIBLE). >>> What is the difference in response times between weblogic and tomcat? >>> ANSWER: FOR PAGES THAT WEBLOGIC SERVES IN 500 ms ON AVERAGE, TOMCAT IS >>> TAKING 17000 ms ON AVERAGE. >>> Maybe your tomcat just perform the job MUCH faster ;-) >>> The thread dump you posted in another reply indicates that your >>> tomcats are idling. >>> Do you have manager installed? NO. With manager you could monitor active >>> connections and requests. >>> Do you have probe installed? YES, WE HAVE IT INSTALLED. A LOT OF THREADS >>> ARE IDLING IS WHAT WE OBSERVE. >>> Finally, if you really want to profile your application, do you have >>> moskito installed? :-) (the one i initially wrote >>> -moskito.anotheria.net, therefore the shameless advertisement). :-) SURE >>> WILL GIVE IT A TRY [TIME PERMITTING]. >>> >>> In the last case I can offer you to provide support getting it running. >>> WILL LET YOU KNOW IF WE ARE INTERESTED. THANKS. >>> >>> regards >>> Leon >>> >>> On Fri, Sep 2, 2011 at 3:50 PM, Talha Fazal <tfa...@credera.com> wrote: >>>> Plz. see my answers below in UPPERCAPS. >>>> >>>> Thanks! >>>> >>>> -----Original Message----- >>>> From: Leon Rosenberg [mailto:rosenberg.l...@gmail.com] >>>> Sent: Friday, September 02, 2011 8:35 AM >>>> To: Tomcat Users List >>>> Subject: Re: Tomcat Performance Turning. >>>> >>>> Hello Talha, >>>> >>>> from a quick glance at your post, do you have the same 30 max threads >>>> limit in weblogic? YES. >>>> Because sending 525 users through 2x30=60 max threads seems a little >>>> bottlenecky. I FORGOT TO MENTION THAT ON EACH TOMCAT SERVER, WE ARE >>>> RUNNING 3 INSTANCES OF TOMCAT ON EACH OF OUR TOMCAT SERVERS (2 SERVERS), >>>> THUS 2*3*30=180 maxThreads. WE EVEN INCREASED THE maxThreads TO 120 FOR >>>> EACH INSTANCE (thus 120*6=720 maxThreads), AND MATCHED maxActive IN JNDI >>>> RESOURCE TO 120, BUT WE STILL DIDN'T SEE CONSIDERABLE IMPROVEMENT. THE CPU >>>> USAGE INCREASED FROM 7% to 10%. >>>> >>>> Which software are you using to produce the load? LOADRUNNER. Does it >>>> keepalive the connections? YES. >>>> >>>> regards >>>> Leon >>>> >>>> On Fri, Sep 2, 2011 at 3:31 PM, Talha Fazal <tfa...@credera.com> wrote: >>>>> Background: >>>>> >>>>> We have a moderately high traffic web application (between 8 to 21 >>>>> million hits/day) running Apache to serve static content (also to load >>>>> balance and create a DMZ) and Weblogic to serve dynamic content (Struts >>>>> 1.1 based Java web application). >>>>> We are trying to replace Weblogic with Tomcat and we have ported our code >>>>> to work with Tomcat. All works well in Tomcat in the DEV, QA, and STAGING >>>>> environment as long as there is no real load. >>>>> >>>>> The Issue - Load Testing: >>>>> >>>>> In our staging environment for load testing, when we run the load test >>>>> using 525 concurrent users, the app doesn't perform at all. The CPU usage >>>>> (on Apache and Tomcat Servers) hovers between 7% to 8%. The database >>>>> server CPU usage is also between 4 and 5%. >>>>> >>>>> Setup for Load Testing: >>>>> >>>>> We have setup 2 apache web servers (4 Quad Processors i.e. 16 CPUs each), >>>>> 2 Tomcat (version 6.0.29) servers (4 Quad Processors i.e. 16 CPUs each). >>>>> Each server has 32 Gb ram. We are using AJP 1.3 to connect Tomcat and >>>>> Apache. >>>>> >>>>> Mentioned below is the version information: >>>>> Apache Version 2.2.14 (with mod_jk module) >>>>> Tomcat: 6.0.29 >>>>> Database: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - >>>>> 64bits >>>>> Connection Pool: DBCP. >>>>> >>>>> Mentioned below are connector settings in conf/server.xml: >>>>> >>>>> <Connector >>>>> address="stagingTCserver01" >>>>> backlog="300" >>>>> connectionTimeout="60000" >>>>> enableLookups="false" >>>>> maxPostSize="2097152" >>>>> maxSpareThreads="10" >>>>> maxThreads="30" >>>>> minSpareThreads="5" >>>>> port="8006" >>>>> protocol="AJP/1.3" >>>>> tcpNoDelay="true" >>>>> /> >>>>> >>>>> Mentioned below are the settings for JNDI resource configured in >>>>> conf/context.xml: >>>>> >>>>> <Resource >>>>> name="jdbc/onlinedb" >>>>> auth="Container" >>>>> type="javax.sql.DataSource" >>>>> driverClassName="oracle.jdbc.OracleDriver" >>>>> url="jdbc:oracle:thin:@192.168.12.10:1521:WEBDB" >>>>> initialSize="1" >>>>> maxActive="30" >>>>> minIdle="1" >>>>> maxIdle="5" >>>>> maxWait="300000" >>>>> poolPreparedStatements="true" >>>>> maxOpenPreparedStatements="300" >>>>> validationQuery="SELECT 1 FROM BB_DUAL" >>>>> testOnBorrow="true" >>>>> validationInterval="10000" >>>>> testWhileIdle="true" >>>>> /> >>>>> >>>>> JVM Parameters: >>>>> >>>>> -Xms512m -Xmx2048m >>>>> -XX:PermSize=128m -XX:MaxPermSize=256m >>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/WL2TC/app/ >>>>> -verbose:gc -Xloggc:/logs/WL2TC/app/WL2TC_1-gc.log >>>>> >>>>> It must be noted that Weblogic setup performs very well using similar >>>>> settings. >>>>> >>>>> Garbage Collection: While the load test is running, Garbage collection >>>>> works just fine i.e. Young GC occurring every 2-3 minutes and takes less >>>>> than half a second. Full GC occurs every hour and takes a little over 2 >>>>> seconds. >>>>> >>>>> Any tips/pointers will be greatly appreciated. >>>>> >>>>> Talha. >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
