Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/23/13 11:11 PM, Chirag Dewan wrote: Chris, The profiler shows very high CPU utilization in Tomcat threads. As I said before a lot of time was spent during the ResponseFacade.setContentType() method call. That doesn't tell the whole story but more or less the high utilization is mainly in the tomcat threads. Unless you are starting your own threads, all threads are Tomcat threads. The threading doesn't matter: it's the method calls that matter. So, profiling still shows that RequestFaade.setContentType is still taking the majority of time? By how much? Can you give us some data? Just because RequestFacade.setContentType is taking most of the time doesn't mean that it's actually a bottleneck. But... recent changes to that particular method will definitely take more time than it has in the past. Can you give us some usage data? Since you are profiling a simple webapp, can you post the code somewhere? I would hope that simple content-types would sail right through that method, while complex content-types (with parameters, etc.) will take longer. The thing which puzzles me the most is I am able to scale the req/sec with multiple clients on Tomcat 6 and the same doesn't happen on Tomcat 7? Can it be the JVM? Would it help trying with a latter version of JVM? If you are using the same JVM for both Tomcat 6 and Tomcat 7, it's unlikely to be the JVM. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRn382AAoJEBzwKT+lPKRYXecP/1E5lg/CkoHCCw61V9QMaNSX 0NzxHtGRc3JQ+Zm9kQAGZpP9rSUEJH5TfFkmn7zYrTfadObA2HmInJnjQljjDQch kAYz9sPmkDDPho82RKBxXUh6IGxikByKl4iEMwPoCdfj22iJT5hxr0kI16RQhjEa VWBFoBE/lYZv7pnNwEgNBKfRP0iKIyFf0kvPXjNJCn4L9MYuCkKTAlXdDdNHlv3N LLK8eMTraOvGvqYd6IGj85bhs47l/uV9PASlIWqsCJB4SpeI/fbSKH7mhsilK316 x63uj/Qw/PAU4k037+QDu+cG3X6eVMnYHkQfIGmRjyzmHw3yIujWsnf0qUXLjh5A F4Vz2+y/iNCNkww00aaxrMZphEmiWZpY2eG3GqMstNU1IQYFdRT+/+teBLY9CDkr uzXaE1xcpZmZMiinLhBuZEaWd4CH/t8i6znjfQeHBCwhwKH+k/A4Obu/EPN7I29/ dPYzxmlCdEXl4x9UTk5hLcImSJMTukrWbnWL9B6KFuywNab5/9W8GGnd0Jv2dE8P ovh0zDfmrQomxE1Vep7W3VksPZyHNwV+z07pzZ/YUKoYTDuTNvRW8QlFPkpnkqzo xNrfsAA1se+3puEkeH+tun6t/243r9CoVIFw43gUivUzN1gUqaJm3f27AJwvDVa7 0o8zCFm5Qq5Wfla2HdBW =dpKM -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 24/05/2013 15:54, Christopher Schultz wrote: Chirag, On 5/23/13 11:11 PM, Chirag Dewan wrote: Chris, The profiler shows very high CPU utilization in Tomcat threads. As I said before a lot of time was spent during the ResponseFacade.setContentType() method call. That doesn't tell the whole story but more or less the high utilization is mainly in the tomcat threads. Unless you are starting your own threads, all threads are Tomcat threads. The threading doesn't matter: it's the method calls that matter. So, profiling still shows that RequestFaade.setContentType is still taking the majority of time? By how much? Can you give us some data? Just because RequestFacade.setContentType is taking most of the time doesn't mean that it's actually a bottleneck. But... recent changes to that particular method will definitely take more time than it has in the past. Make sure you are using the latest 7.0.x release. Testing anything is fairly pointless. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/23/13 11:11 PM, Chirag Dewan wrote: As I said before a lot of time was spent during the ResponseFacade.setContentType() method call. That doesn't tell the whole story but more or less the high utilization is mainly in the tomcat threads. Sorry, I forgot to mention another important point: the fact that ResponseFacade.setContentType takes a relatively long time when accessing a trivial resource is not that important. Instead, what's important is how much that call takes relative to your typical transaction time. Obviously, you want to limit Tomcat's overhead as much as possible. But saying that Tomcat 7 is unacceptable because it gets you only 70k req/sec versus Tomcat 6's 120k req/sec is not exactly fair. What you really ought to do it benchmark your webapp. If your webapp was only able to handle 100 tps on Tomcat 6, then it's not suddenly only going to process 50 tps on Tomcat 7. Also, you've been throwing-around a lot of different numbers so it's tough to keep them straight. 70k on Tomcat 7 versus 120k on Tomcat 6, but Tomcat 6 only gives you 70k req/sec with 25-30% CPU utilization. But also Tomcat 6 with 48k req/sec for each of 3 clients (that's 150k req/sec overall) and you are calling it 120k req/sec? I'm totally lost with all the random numbers you are tossing out here without a) providing any configuration information such as a Connector b) providing any code or c) providing any rich information from a recognizable load-test. You have mentioned both JMeter and HttpClient (which I think you discarded because it was giving you bad performance). There's also ApacheBench which is freely available and comes with Apache httpd. All of these tools should produce some kind of report. For example, if I run ab against my Tomcat 7.0.40 instance just requesting /index.html as fast as possible long enough to get a CPU usage number in 'top', I get this report: Benchmarking 127.0.0.1 (be patient) Server Software:Apache-Coyote/1.1 Server Hostname:127.0.0.1 Server Port:8217 Document Path: /webapp/index.html Document Length:3460 bytes Concurrency Level: 10 Time taken for tests: 85.627 seconds Complete requests: 49046 Failed requests:140 (Connect: 0, Receive: 70, Length: 70, Exceptions: 0) Write errors: 0 Total transferred: 181554032 bytes HTML transferred: 169456960 bytes Requests per second:572.79 [#/sec] (mean) Time per request: 17.458 [ms] (mean) Time per request: 1.746 [ms] (mean, across all concurrent requests) Transfer rate: 2070.61 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:02 42.9 01609 Processing: 0 14 326.4 18693 Waiting:01 1.2 1 79 Total: 0 16 329.1 18693 top showed that Tomcat was taking wildly-varying amounts of CPU: sometimes 50%, sometimes less than 1%, etc. I never saw it go above 50%. I suspect I've hit some kind of I/O limit, somewhere. Running again with HTTP keepalives enabled, I have a different report: Benchmarking 127.0.0.1 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache-Coyote/1.1 Server Hostname:127.0.0.1 Server Port:8217 Document Path: /webapp/index.html Document Length:3460 bytes Concurrency Level: 10 Time taken for tests: 57.293 seconds Complete requests: 100 Failed requests:0 Write errors: 0 Keep-Alive requests:990005 Total transferred: 3711950025 bytes HTML transferred: 346000 bytes Requests per second:17454.15 [#/sec] (mean) Time per request: 0.573 [ms] (mean) Time per request: 0.057 [ms] (mean, across all concurrent requests) Transfer rate: 63270.43 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 0.0 0 12 Processing: 01 0.6 1 57 Waiting:01 0.6 0 57 Total: 01 0.6 1 57 WARNING: The median and mean for the waiting time are not within a normal deviation These results are probably not that reliable. Percentage of the requests served within a certain time (ms) 50% 1 66% 1 75% 1 80% 1 90% 1 95% 1 98% 1 99% 2 100% 57 (longest request) Top reported that Tomcat was taking somewhere between 550-600% CPU. (This is a 4-core hyperthreaded CPU so I have 8 logical cores. 'ab' was taking about 100% CPU so I think 600% CPU means it was roughly pegging 6 of my logical
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Chris, Sorry I should have posted the data first. I probably missed the most important part of a load test. I will do it shortly. And I am not using Jmeter now,I am using an http client for load test. I am testing it on Solaris x86 server 64bit JVM. And i have collected the samples for Tomcat 6.0.18,0.37,7.0.30 and 6.0.40. I do not want to use ab because of factor that my benchmark in tomcat 6 were established on a custom http client. I do not have the exact data with me right now. Will surely post it asap. Thanks. Sent from Yahoo! Mail on Android
[OT] ab and load testing (was: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7)
On 24.05.2013 17:54, Christopher Schultz wrote: Top reported that Tomcat was taking somewhere between 550-600% CPU. (This is a 4-core hyperthreaded CPU so I have 8 logical cores. 'ab' was taking about 100% CPU so I think 600% CPU means it was roughly pegging 6 of my logical cores. Roughly 30% system, 60% user CPU usage.). The 100% CPU for ab indicates that ab was your bottleneck. AFAIK ab is single threaded async, so it can handle multiple parallel connections but will max out on one consumed CPU core. If you want to go further, you would have to start multiple ab in parallel. Since you are close to a CPU saturated system, in this specific case you might not get much further, but 17454 requests/sec isn't that bad either :) Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Hi All, I tested Tomcat 7.0.40 with Solaris x86. It gave better CPU performance than Tomcat 7.0.30 . It was able to handle 70K requests at 45% CPU utilization which is still behind my bench mark of 70k requests at 25-30% utilization with Tomcat 6. I also tested with Tomcat 6.0.37 and the results were more or less same as Tomcat 6.0.18 One thing I observed,I ran 3 clients pointing to 3 different static HTML pages. With Tomcat 6(both 6.0.18 and 6.0.37) each client gave me a concurrency of 48k req/sec at 50-55% CPU utilization. However when I ran the same test case for Tomcat 7(7.0.40 and 7.0.30) the req/sec were divided b/w the clients and the total concurrency was consistent around 70k with 45% utilization. Now no matter how many clients I run,the req/sec is constant. This is a bit strange. What could possibly be the reason? I am not sure I would be able to upgrade to Tomcat 7 under these circumstances. Thanks. From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, 22 May 2013 7:30 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/21/13 11:03 PM, Chirag Dewan wrote: I was monitoring the CPU utilization specifically. I can compromise on 1 less transactions/sec,but 80-90% utilization is not good. While it's nice to reduce resource utilization as much as possible, why would you want your server to run at 25% CPU all the time? That's a waste of 75% of the CPU available. The test you are performing sounds like a peak-load test (100 simultaneous connections is pretty decent load). At peak load, you have 10-20% breathing room before your CPU can't keep up. Sounds like you have an appropriately-sized server for the type of load you expect. Do you have some kind of performance benchmarks that you have to hit? I have configured 1000 max threads in the connector,and I am testing with 100 concurrent threads. So you have 10 times the number of threads available than necessary. Did you do that simply to make sure that the thread-pool wouldn't be the problem? What kind of connector are you using? Re-reading the thread it looks like you are using the BIO (default) connector. I'm not sure switching to NIO or APR would help raw throughput unless you are using SSL (switch to APR) or have lots of dropped keepalive connections (switch to NIO - or APR?). Well I tested a sample html page with Tomcat 6.0.18 and 7.0.30 on Solaris x86 server. The req/sec were almost the same for both,but CPU utilization for Tomcat 6 was 23-27% and for Tomcat 7 it was 80-90% consistently. What about Tomcat 6.0.37? On the other hand on a Linux system,the req/sec were a little higher in Tomcat 7 say 65k to 55k on tomcat 6 and the CPU utilization was the same for both 6 and 7. ... and what was that CPU utilization? My Jprofiler stack trace on Solaris is a lot different. As far as I have observed,for Tomcat 7 the stack Trace leads me to ResponseFacade.setContentType,which was not the behaviour in Tomcat 6. Can that be a bottleneck? See Mark's message: there have been some changes to how Tomcat performs a setContentType -- because the string actually needs to be parsed to see if there is a character encoding hiding in there, and the output writer may need it's encoding changed. You can see in the changelog (http://tomcat.apache.org/tomcat-7.0-doc/changelog.html) this entry under 7.0.33, which I think is the relevant part: The HTTP header parser added to address 52811 has been removed... Take a look at the references provided if you are interested in the details. Otherwise, just upgrade to 7.0.40 and try again. Or is there something platform dependent that I should take care of? There could be a bug in one of the JVMs, but this is fairly well-exercised code that you are talking about. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRnM+NAAoJEBzwKT+lPKRYDo4P/2LGsApJTBj39lPKdshAYXba DwDWIllFM+fn/XRLsx2fTinBLPupWoGbGrPoVOA7xTTqrJVddJCfwbbmiS2RvcB5 kPMNn+baAWMN3f4z/5fVkXwVhuyfRNKMFNzL64N3hjuqhzMqQGBvoaUbKMq27bzt VzQiLiEC5mUJ4VsnUj+h4yWCFPv2JT7Q4tifneAX3F3ouOI6KqJ7y30LX2qz4xPp 4f/KDgQwMTxrsdDe2R3+dmBFZgo2sHuZZUggGCvTADmjAaPgikAWAO5Yov2tPVhf ZRLrn6s6VAQpRDPN6jNQbogWWw/zakDf8zPrTSjr/BQRrhF7+eRyr3/jWlA9UOkd rs/WIXm6URZjhbzdO7h0qkPOziSw9cekUt/cVUigd9N+Mgyp0YcDw8wtg6Hx0PJC Ig/nvuOB3ePxP01EPc0hXg3fw6GlwTSS0H5GRox0n5cvR83lDtGrzCvqZe+py6z4 JaKtA7FhcYcZrqZiiGMNVYsMcBL5CEr4gQztCbQscfltnTjCIQoM7BclQ+VMHDQI SfLEsMfNBix6ap88H335RAyGUyMbz1kWMtBv7+GvzEoBQ6GDfk39iDu2rLfFLiMi calZtW9asdFFQwpe87hdadPToOgFuyyAIm7EZpSWF69kh5nM6DXSEBGOhjCrc9OB QGR+cvRCWYnodpC4YZhm =NxRr -END PGP SIGNATURE
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 5/23/2013 4:53 AM, Chirag Dewan wrote: Hi All, I tested Tomcat 7.0.40 with Solaris x86. It gave better CPU performance than Tomcat 7.0.30 . It was able to handle 70K requests at 45% CPU utilization which is still behind my bench mark of 70k requests at 25-30% utilization with Tomcat 6. I also tested with Tomcat 6.0.37 and the results were more or less same as Tomcat 6.0.18 One thing I observed,I ran 3 clients pointing to 3 different static HTML pages. With Tomcat 6(both 6.0.18 and 6.0.37) each client gave me a concurrency of 48k req/sec at 50-55% CPU utilization. However when I ran the same test case for Tomcat 7(7.0.40 and 7.0.30) the req/sec were divided b/w the clients and the total concurrency was consistent around 70k with 45% utilization. Now no matter how many clients I run,the req/sec is constant. This is a bit strange. What could possibly be the reason? Maybe you're saturating the bandwidth or maxing out some other resource not connected to the tomcats? I am not sure I would be able to upgrade to Tomcat 7 under these circumstances. Thanks. From: Christopher Schultzch...@christopherschultz.net To: Tomcat Users Listusers@tomcat.apache.org Sent: Wednesday, 22 May 2013 7:30 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/21/13 11:03 PM, Chirag Dewan wrote: I was monitoring the CPU utilization specifically. I can compromise on 1 less transactions/sec,but 80-90% utilization is not good. While it's nice to reduce resource utilization as much as possible, why would you want your server to run at 25% CPU all the time? That's a waste of 75% of the CPU available. The test you are performing sounds like a peak-load test (100 simultaneous connections is pretty decent load). At peak load, you have 10-20% breathing room before your CPU can't keep up. Sounds like you have an appropriately-sized server for the type of load you expect. Do you have some kind of performance benchmarks that you have to hit? I have configured 1000 max threads in the connector,and I am testing with 100 concurrent threads. So you have 10 times the number of threads available than necessary. Did you do that simply to make sure that the thread-pool wouldn't be the problem? What kind of connector are you using? Re-reading the thread it looks like you are using the BIO (default) connector. I'm not sure switching to NIO or APR would help raw throughput unless you are using SSL (switch to APR) or have lots of dropped keepalive connections (switch to NIO - or APR?). Well I tested a sample html page with Tomcat 6.0.18 and 7.0.30 on Solaris x86 server. The req/sec were almost the same for both,but CPU utilization for Tomcat 6 was 23-27% and for Tomcat 7 it was 80-90% consistently. What about Tomcat 6.0.37? On the other hand on a Linux system,the req/sec were a little higher in Tomcat 7 say 65k to 55k on tomcat 6 and the CPU utilization was the same for both 6 and 7. ... and what was that CPU utilization? My Jprofiler stack trace on Solaris is a lot different. As far as I have observed,for Tomcat 7 the stack Trace leads me to ResponseFacade.setContentType,which was not the behaviour in Tomcat 6. Can that be a bottleneck? See Mark's message: there have been some changes to how Tomcat performs a setContentType -- because the string actually needs to be parsed to see if there is a character encoding hiding in there, and the output writer may need it's encoding changed. You can see in the changelog (http://tomcat.apache.org/tomcat-7.0-doc/changelog.html) this entry under 7.0.33, which I think is the relevant part: The HTTP header parser added to address 52811 has been removed... Take a look at the references provided if you are interested in the details. Otherwise, just upgrade to 7.0.40 and try again. Or is there something platform dependent that I should take care of? There could be a bug in one of the JVMs, but this is fairly well-exercised code that you are talking about. - -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 5/23/13 7:40 AM, David kerber wrote: On 5/23/2013 4:53 AM, Chirag Dewan wrote: Hi All, I tested Tomcat 7.0.40 with Solaris x86. It gave better CPU performance than Tomcat 7.0.30 . It was able to handle 70K requests at 45% CPU utilization which is still behind my bench mark of 70k requests at 25-30% utilization with Tomcat 6. I also tested with Tomcat 6.0.37 and the results were more or less same as Tomcat 6.0.18 One thing I observed,I ran 3 clients pointing to 3 different static HTML pages. With Tomcat 6(both 6.0.18 and 6.0.37) each client gave me a concurrency of 48k req/sec at 50-55% CPU utilization. However when I ran the same test case for Tomcat 7(7.0.40 and 7.0.30) the req/sec were divided b/w the clients and the total concurrency was consistent around 70k with 45% utilization. Now no matter how many clients I run,the req/sec is constant. This is a bit strange. What could possibly be the reason? Maybe you're saturating the bandwidth or maxing out some other resource not connected to the tomcats? Or hitting the limit of what Tomcat can actually handle in that environment. I would expect that, at some point, you'll hit a wall where (total) req/sec cannot go any higher no matter which way you adjust the attack (concurrency versus volume-per-client). Are you saying that with 2 clients (c=2) you can get 70K req/sec /per client/ and then also with c=50 you can still get 70K req/sec /per client/ meaning that Tomcat is serving 3.5M req/sec overall? That seems ... unlikely. I am not sure I would be able to upgrade to Tomcat 7 under these circumstances. Are you saying that Tomcat 7 can't serve your peak load requirements, or that you have some paper benchmarks you want to hit and since you can't hit them, Tomcat 7 is not an option? I'm still not sure why you are terribly concerned with CPU utilization. If you've got the CPU, why not use it? An idle machine is one that is costing you money. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRnjoYAAoJEBzwKT+lPKRYmPMQALVEineIFqQXf6Wx4EaueozR OtwvQJ/bKIS9//JzUfAt4PmnEykfu0XD4xwzoumtCdLLZjIQNsHXc1M6WCvd+kUU ke4qgRdVJiK58ZKHKNqjEJbknLOm+qHOyNQMzKP3w84lt9kx4Vr/CayhD6YF+pGw o1aDlcWnjnwjJXA4+UWVxmLclFzyby3sC+iMd+Z+zSA5es5vDIU5zlyWDCbWVFv7 bbIEAU81StdSSugalPsSMtxXMUXpIeirOmIvdtJI2pLiuUI6RK0HiMW8IpPzjN3n yxERCLLSh+tupBk8Sb5PozCu1t7GNa24rvcmqP3rH2mFQ8oRVYmu1P0KZqTjL/yw zPBxFip1vTcGOZxiFPA7c9XafW983jcgK2r1Jqxg23R4+sxBeIaCyDuRK9TBtQTn KfgqTUG05SQSoLr67a0+VOjEQZzAU9G2cvAGAYBJU8RJcK1wAn0nO4Xg2Mz9T3fi t2kJCzeNV01SUPkJL5iAGdVO5HVHDvij1CO1OD6O0IqSDmL1RxQw75MyygEqc9GV U41lg5VtbtM3RGLdamJzmIZwSiczzmMW5XIvDFQ9Dc+gubAkScSdh99259KvTCYM 3wEMCBplx0q3jeyBTmNfZ0PvM7GE656GDSLr/KY1QuajBwP97rXsvFx1SDlv6hw7 OG9G6kOYVZQOxe8+mYLO =zEBg -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Chris, With c=1 it is 70k req/sec and with c=2 it is 35k in both i.e the total req/sec cannot be scaled beyond 70k. With Tomcat 6 it is 60k in both clients i.e total of 120k. I do not expect more than 75k req/sec being served by Tomcat 7,but its the benchmark set by Tomcat 6 which I can't overlook. As I said I wouldn't have mind 80-90 % utilization on peak load,but its the already running server performance which I am compelled to follow or even carry out more. Thanks. Sent from Yahoo! Mail on Android
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/23/13 1:21 PM, Chirag Dewan wrote: With c=1 it is 70k req/sec and with c=2 it is 35k in both i.e the total req/sec cannot be scaled beyond 70k. With Tomcat 6 it is 60k in both clients i.e total of 120k. I do not expect more than 75k req/sec being served by Tomcat 7,but its the benchmark set by Tomcat 6 which I can't overlook. As I said I wouldn't have mind 80-90 % utilization on peak load,but its the already running server performance which I am compelled to follow or even carry out more. What do you see when you attach a profiler to Tomcat while you run your load tests against it? (Now that you have removed your own webapp from the mix and are just fetching trivial resources). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRnpAFAAoJEBzwKT+lPKRY91AQAJicVT7nLNBzr4m7eqOs6sRn R+hCNT37A0HWS8+A8Ns52FyCUUmOljrFVP8LcwA1J0IXFxxI1GvCGNdL7y8248dF cWKyyMYUpmiGbwRBcz0XTvxJoGLwo55Wv7pnYsdecU7d93epXcPSZTQHM3zI82ZB TEShV/AHigcnA9mEe8NV1XgLzCZclh9mjfP2uIhhBHvNkoK3gX0oMey8xK9lezlJ ArGKy1RUNsvnWaDvmFC5o5FQyh8CPDnxXalGgDKnjxFHQfUOtCXaFdWdgeiPL6xn TPiuWIEdqNXERLAjY9IjXzoXpBkxRcYPEwN4V3/tHZapw6C7M9OmcmuXMfhF5UEP G4e+nhcy2CtdUum38j4g1LLEB06QRmc1DIt6czb3kqKQR3FTBpRSkBdWEjJhZnxU JVRIYiK8rOqxXygJcga0VlI5xSIzHh2JK7qlT6rc+zqWQTHXOFkmxyGMfjb+hx+i Mofp0lrO9aczFopb/C3uF7gOAb5c0hhhf1Smxvy9nCl2YscA+BarhWdwiOeEGd+6 mNTK/unQIZLJI3Sqb8xgaaGoa5FEvfoaRptweiqON76hNZGrFDxCnQeG9h6W12uW CGQfEWnbgmWkao/ZrIGYkxmRQNd7n4w2jEfzxdD40yrDX+jyQ7VbWrKkX/m/0K+b 7h8osL0Vaqk+GtaWfuPA =cRWt -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Chris, The profiler shows very high CPU utilization in Tomcat threads. As I said before a lot of time was spent during the ResponseFacade.setContentType() method call. That doesn't tell the whole story but more or less the high utilization is mainly in the tomcat threads. The thing which puzzles me the most is I am able to scale the req/sec with multiple clients on Tomcat 6 and the same doesn't happen on Tomcat 7? Can it be the JVM? Would it help trying with a latter version of JVM? Thanks. From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, 24 May 2013 3:24 AM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/23/13 1:21 PM, Chirag Dewan wrote: With c=1 it is 70k req/sec and with c=2 it is 35k in both i.e the total req/sec cannot be scaled beyond 70k. With Tomcat 6 it is 60k in both clients i.e total of 120k. I do not expect more than 75k req/sec being served by Tomcat 7,but its the benchmark set by Tomcat 6 which I can't overlook. As I said I wouldn't have mind 80-90 % utilization on peak load,but its the already running server performance which I am compelled to follow or even carry out more. What do you see when you attach a profiler to Tomcat while you run your load tests against it? (Now that you have removed your own webapp from the mix and are just fetching trivial resources). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRnpAFAAoJEBzwKT+lPKRY91AQAJicVT7nLNBzr4m7eqOs6sRn R+hCNT37A0HWS8+A8Ns52FyCUUmOljrFVP8LcwA1J0IXFxxI1GvCGNdL7y8248dF cWKyyMYUpmiGbwRBcz0XTvxJoGLwo55Wv7pnYsdecU7d93epXcPSZTQHM3zI82ZB TEShV/AHigcnA9mEe8NV1XgLzCZclh9mjfP2uIhhBHvNkoK3gX0oMey8xK9lezlJ ArGKy1RUNsvnWaDvmFC5o5FQyh8CPDnxXalGgDKnjxFHQfUOtCXaFdWdgeiPL6xn TPiuWIEdqNXERLAjY9IjXzoXpBkxRcYPEwN4V3/tHZapw6C7M9OmcmuXMfhF5UEP G4e+nhcy2CtdUum38j4g1LLEB06QRmc1DIt6czb3kqKQR3FTBpRSkBdWEjJhZnxU JVRIYiK8rOqxXygJcga0VlI5xSIzHh2JK7qlT6rc+zqWQTHXOFkmxyGMfjb+hx+i Mofp0lrO9aczFopb/C3uF7gOAb5c0hhhf1Smxvy9nCl2YscA+BarhWdwiOeEGd+6 mNTK/unQIZLJI3Sqb8xgaaGoa5FEvfoaRptweiqON76hNZGrFDxCnQeG9h6W12uW CGQfEWnbgmWkao/ZrIGYkxmRQNd7n4w2jEfzxdD40yrDX+jyQ7VbWrKkX/m/0K+b 7h8osL0Vaqk+GtaWfuPA =cRWt -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 22/05/2013 04:03, Chirag Dewan wrote: My Jprofiler stack trace on Solaris is a lot different. As far as I have observed,for Tomcat 7 the stack Trace leads me to ResponseFacade.setContentType,which was not the behaviour in Tomcat 6. Can that be a bottleneck? Or is there something platform dependent that I should take care of? Try again with 7.0.40. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/21/13 11:03 PM, Chirag Dewan wrote: I was monitoring the CPU utilization specifically. I can compromise on 1 less transactions/sec,but 80-90% utilization is not good. While it's nice to reduce resource utilization as much as possible, why would you want your server to run at 25% CPU all the time? That's a waste of 75% of the CPU available. The test you are performing sounds like a peak-load test (100 simultaneous connections is pretty decent load). At peak load, you have 10-20% breathing room before your CPU can't keep up. Sounds like you have an appropriately-sized server for the type of load you expect. Do you have some kind of performance benchmarks that you have to hit? I have configured 1000 max threads in the connector,and I am testing with 100 concurrent threads. So you have 10 times the number of threads available than necessary. Did you do that simply to make sure that the thread-pool wouldn't be the problem? What kind of connector are you using? Re-reading the thread it looks like you are using the BIO (default) connector. I'm not sure switching to NIO or APR would help raw throughput unless you are using SSL (switch to APR) or have lots of dropped keepalive connections (switch to NIO - or APR?). Well I tested a sample html page with Tomcat 6.0.18 and 7.0.30 on Solaris x86 server. The req/sec were almost the same for both,but CPU utilization for Tomcat 6 was 23-27% and for Tomcat 7 it was 80-90% consistently. What about Tomcat 6.0.37? On the other hand on a Linux system,the req/sec were a little higher in Tomcat 7 say 65k to 55k on tomcat 6 and the CPU utilization was the same for both 6 and 7. ... and what was that CPU utilization? My Jprofiler stack trace on Solaris is a lot different. As far as I have observed,for Tomcat 7 the stack Trace leads me to ResponseFacade.setContentType,which was not the behaviour in Tomcat 6. Can that be a bottleneck? See Mark's message: there have been some changes to how Tomcat performs a setContentType -- because the string actually needs to be parsed to see if there is a character encoding hiding in there, and the output writer may need it's encoding changed. You can see in the changelog (http://tomcat.apache.org/tomcat-7.0-doc/changelog.html) this entry under 7.0.33, which I think is the relevant part: The HTTP header parser added to address 52811 has been removed... Take a look at the references provided if you are interested in the details. Otherwise, just upgrade to 7.0.40 and try again. Or is there something platform dependent that I should take care of? There could be a bug in one of the JVMs, but this is fairly well-exercised code that you are talking about. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRnM+NAAoJEBzwKT+lPKRYDo4P/2LGsApJTBj39lPKdshAYXba DwDWIllFM+fn/XRLsx2fTinBLPupWoGbGrPoVOA7xTTqrJVddJCfwbbmiS2RvcB5 kPMNn+baAWMN3f4z/5fVkXwVhuyfRNKMFNzL64N3hjuqhzMqQGBvoaUbKMq27bzt VzQiLiEC5mUJ4VsnUj+h4yWCFPv2JT7Q4tifneAX3F3ouOI6KqJ7y30LX2qz4xPp 4f/KDgQwMTxrsdDe2R3+dmBFZgo2sHuZZUggGCvTADmjAaPgikAWAO5Yov2tPVhf ZRLrn6s6VAQpRDPN6jNQbogWWw/zakDf8zPrTSjr/BQRrhF7+eRyr3/jWlA9UOkd rs/WIXm6URZjhbzdO7h0qkPOziSw9cekUt/cVUigd9N+Mgyp0YcDw8wtg6Hx0PJC Ig/nvuOB3ePxP01EPc0hXg3fw6GlwTSS0H5GRox0n5cvR83lDtGrzCvqZe+py6z4 JaKtA7FhcYcZrqZiiGMNVYsMcBL5CEr4gQztCbQscfltnTjCIQoM7BclQ+VMHDQI SfLEsMfNBix6ap88H335RAyGUyMbz1kWMtBv7+GvzEoBQ6GDfk39iDu2rLfFLiMi calZtW9asdFFQwpe87hdadPToOgFuyyAIm7EZpSWF69kh5nM6DXSEBGOhjCrc9OB QGR+cvRCWYnodpC4YZhm =NxRr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Chris, I am setting previously running Tomcat 6 performance data as a benchmark. And I am straight forward comparing the results with Tomcat 7 to the previous results. I expect a similar performance,if not better. Actually,the max no. of connections in my case are configurable from a GUI with a maximum of 1000 that a user can set. Hence 1000 max threads would be my peak load case. Well for now I have completely dropped my application out of the scope. I am testing and comparing Tomcat 6 and 7 on the same servers with sample HTML pages. Next in line would be Tomcat 6.0.37 and 7.0.40. I would have to upgrade to minor version of Tomcat 6 otherwise. Thanks. From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, 22 May 2013 7:30 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/21/13 11:03 PM, Chirag Dewan wrote: I was monitoring the CPU utilization specifically. I can compromise on 1 less transactions/sec,but 80-90% utilization is not good. While it's nice to reduce resource utilization as much as possible, why would you want your server to run at 25% CPU all the time? That's a waste of 75% of the CPU available. The test you are performing sounds like a peak-load test (100 simultaneous connections is pretty decent load). At peak load, you have 10-20% breathing room before your CPU can't keep up. Sounds like you have an appropriately-sized server for the type of load you expect. Do you have some kind of performance benchmarks that you have to hit? I have configured 1000 max threads in the connector,and I am testing with 100 concurrent threads. So you have 10 times the number of threads available than necessary. Did you do that simply to make sure that the thread-pool wouldn't be the problem? What kind of connector are you using? Re-reading the thread it looks like you are using the BIO (default) connector. I'm not sure switching to NIO or APR would help raw throughput unless you are using SSL (switch to APR) or have lots of dropped keepalive connections (switch to NIO - or APR?). Well I tested a sample html page with Tomcat 6.0.18 and 7.0.30 on Solaris x86 server. The req/sec were almost the same for both,but CPU utilization for Tomcat 6 was 23-27% and for Tomcat 7 it was 80-90% consistently. What about Tomcat 6.0.37? On the other hand on a Linux system,the req/sec were a little higher in Tomcat 7 say 65k to 55k on tomcat 6 and the CPU utilization was the same for both 6 and 7. ... and what was that CPU utilization? My Jprofiler stack trace on Solaris is a lot different. As far as I have observed,for Tomcat 7 the stack Trace leads me to ResponseFacade.setContentType,which was not the behaviour in Tomcat 6. Can that be a bottleneck? See Mark's message: there have been some changes to how Tomcat performs a setContentType -- because the string actually needs to be parsed to see if there is a character encoding hiding in there, and the output writer may need it's encoding changed. You can see in the changelog (http://tomcat.apache.org/tomcat-7.0-doc/changelog.html) this entry under 7.0.33, which I think is the relevant part: The HTTP header parser added to address 52811 has been removed... Take a look at the references provided if you are interested in the details. Otherwise, just upgrade to 7.0.40 and try again. Or is there something platform dependent that I should take care of? There could be a bug in one of the JVMs, but this is fairly well-exercised code that you are talking about. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRnM+NAAoJEBzwKT+lPKRYDo4P/2LGsApJTBj39lPKdshAYXba DwDWIllFM+fn/XRLsx2fTinBLPupWoGbGrPoVOA7xTTqrJVddJCfwbbmiS2RvcB5 kPMNn+baAWMN3f4z/5fVkXwVhuyfRNKMFNzL64N3hjuqhzMqQGBvoaUbKMq27bzt VzQiLiEC5mUJ4VsnUj+h4yWCFPv2JT7Q4tifneAX3F3ouOI6KqJ7y30LX2qz4xPp 4f/KDgQwMTxrsdDe2R3+dmBFZgo2sHuZZUggGCvTADmjAaPgikAWAO5Yov2tPVhf ZRLrn6s6VAQpRDPN6jNQbogWWw/zakDf8zPrTSjr/BQRrhF7+eRyr3/jWlA9UOkd rs/WIXm6URZjhbzdO7h0qkPOziSw9cekUt/cVUigd9N+Mgyp0YcDw8wtg6Hx0PJC Ig/nvuOB3ePxP01EPc0hXg3fw6GlwTSS0H5GRox0n5cvR83lDtGrzCvqZe+py6z4 JaKtA7FhcYcZrqZiiGMNVYsMcBL5CEr4gQztCbQscfltnTjCIQoM7BclQ+VMHDQI SfLEsMfNBix6ap88H335RAyGUyMbz1kWMtBv7+GvzEoBQ6GDfk39iDu2rLfFLiMi calZtW9asdFFQwpe87hdadPToOgFuyyAIm7EZpSWF69kh5nM6DXSEBGOhjCrc9OB QGR+cvRCWYnodpC4YZhm =NxRr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
It might be weird but its confusing me a lot. I used HttpClient 4.2.1 for Tomcat 6.0.18 and I used the same client with Tomcat 7.0.30,causing the CPU utilization and reduced trans/sec. Now,the same thing I have tested with Apache Jmeter 2.6,and its working fine. Any possible reasons? Thanks. From: Chirag Dewan chirag.dewa...@yahoo.in To: Tomcat Users List users@tomcat.apache.org Sent: Monday, 20 May 2013 8:08 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 I ran my test client on Hello World example servlet on Tomcat 7.0.30. It was 12K req/sec with 80% CPU utilization. The same test case on tomcat 6.0.18 gave me similar req/sec but CPU utilization of 70% . The results on my linux server,with client and server running on the same machine. But all I am doing is comparing the 2 instances in the same environement. I am pretty sure the utilization is due to the server. The Jprofiler shows a lot of logging threads in tomcat consuming a lot of memory. Something I should be careful about? Also I ran an asynchronous servlet with the same client. The trans/sec reduced drastically,but the CPU utilization was only 4% Are there too may threads waiting for connection? Would thread pooling help,if we can? Thanks. From: Mark Thomas ma...@apache.org To: Tomcat Users List users@tomcat.apache.org Sent: Monday, 20 May 2013 1:48 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 On 20/05/2013 06:59, Chirag Dewan wrote: Hi, I have profiled the application using JProfiler,and it seems to me that its my servlet which is taking the majority of time. Though the time is in mili seconds,but I guess since the servlet is code is same as with Tomcat 6.0.18 is it the Servlet 3.0 API which is causing the reduced throughput? If the time is being spent in your servlet then it can't possibly be the Servlet 3.0 implementation causing the delay since that would be time spent inside Tomcat code, not your code. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/20/13 10:38 AM, Chirag Dewan wrote: I ran my test client on Hello World example servlet on Tomcat 7.0.30. It was 12K req/sec with 80% CPU utilization. The same test case on tomcat 6.0.18 gave me similar req/sec but CPU utilization of 70% . The results on my linux server,with client and server running on the same machine. But all I am doing is comparing the 2 instances in the same environement. Running both client and server on the same machine is not a great test if you want to determine the server load by looking at CPU utilization. Were you looking at overall CPU utilization, or just utilization of the Tomcat JVM process? I am pretty sure the utilization is due to the server. The Jprofiler shows a lot of logging threads in tomcat consuming a lot of memory. Something I should be careful about? Can you be more specific? AFAIK, there are no logging threads in Tomcat at all. Also I ran an asynchronous servlet with the same client. The trans/sec reduced drastically,but the CPU utilization was only 4% I would expect a small reduction in trans/sec when using async, but nothing I would describe as drastic. Can you give us some numbers? Are there too may threads waiting for connection? Would thread pooling help, if we can? I think you mean too many connections waiting for a thread instead of the other way around. Threads are already pooled. What are your Connector configuration details, and what are the parameters for your load test? If you only have 10 connections configured, launching 10,000 clients isn't exactly a fair test. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRm5UNAAoJEBzwKT+lPKRYKoEQAJeWY07c4jr1prAqvdrOyR4X X9CaWrZ4DwvsKuSn2NtZgXqf5lcLhsDzgP3Ej6cNhBx7WcK6zy8zpTZ2RdVo+YOB 1fGfdpZ6rMODT3i2UI8mAQmbG54NwA4C40ShsTpYVtdXb4GfAX+81j8HqtClr5Tf NEBq2xu4LJDc0ajE4mh+Vyr3M3wUG5CEsQAKhj3Owddtus6sltbrBs8116ZzcpQt 10COnt/e08ikyUVuAYxGZPvQ+rwhlFWuvTf0NlG9/oZSDxSb19KcdZ+vMi2HTNQr McQpVxd2RtqpTgM2NJ3uTS6rQ7R4I1moP96DOgNrU6jS2u5sJVoH98weq4AU4uxa 5sXJG4N4eaVsaFE9m2Nio3zgXZDObmF2jh1df79d7TzHkcIJHjtbZ9VHf1gLVqoW v2pWVU+7di6MiDg7P1Y8y2pXwYYF13ZVx6uozyfv6hGuGXDZkMmmg0QatpKguVXl sO3idan5klic3RJJg2zduq6PUlJZyqJJulqJas1DoUHARo6pSDkZWy1hYZQC9BGT dHgmxrA4Jo5Npq1VpiNMCXl/wCT2vKdsseqsCEQN4EIvgJWuZ5l8NkZtceQTPXtE 4CEcsSd83hIDmichkhVw3PoNHKSSFGwdRRI5sEXqW1Vv88Vj7ldtD8518vJUoQLu s2xIFlQjkJKW/W46Y+on =2P5W -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/21/13 6:08 AM, Chirag Dewan wrote: It might be weird but its confusing me a lot. I used HttpClient 4.2.1 for Tomcat 6.0.18 and I used the same client with Tomcat 7.0.30,causing the CPU utilization and reduced trans/sec. Now,the same thing I have tested with Apache Jmeter 2.6,and its working fine. Any possible reasons? JMeter is actually a load-testing product and much more appropriate to use than a hand-rolled client using HttpClient. I would abandon your hand-rolled load-test and use JMeter instead. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRm5VIAAoJEBzwKT+lPKRYrMoP/jWKRWXHKEw4Bp2XTClhqmY9 fxvuu9HshuEjP4CJvEoDSoTOXAarLu3aiLg5HvUO/bnzdJKqy3MEn6d6EKL1reb4 5LyqbBnhFAALf/VnnSnyDuT/TAvp5QPScq0FJN1osweT91F/xYilTu2MBKXC4TUL 2swtp9VwPty6oNH+wI71b6Kw/anwS+5+9zmB/Ala4OnoUDcInGu2ox2oV3zcCgeC xdveiDJ2+QVnoNeUefwi4XbJcluYKaWfxhs851mtL567Pi66l4puvDz2qAdIHQB/ ZFBPE7iJCd0utVi9gFYBvGTYAcMlYEK8Akul+6+iv6hx2eYh8o+IlbIoMFWIshPS voaMh1rFmRmA1S+q8trSRkCn8x5j0yhDAHD9tWvQqbUGe/RFXMvxxohRZAjsUdFq zn3S5HHW8qNSst/lSHSHDbWOWxETTd9VlQsC7QZnJKkxW7uag6HaIEgCoqncIAH6 +y3QhX1Wwj/lF1uTpuYQFHZwJhRYTVDDhEQvIfn5YE11wSQtsN0ZJhlazpN5FfgF PRhj+u4sx3ZvHELd8OLS9Z74fUVtPyxFFCRYbZRMVA/NGpDGYGswLH+kBVTNUfb+ WKO0PzrfnmqPAQ0TWziFMf4WuNpvcmsYwcpWenYgXLYCAsWTSE1t59byponxsX1q MU1rl/pqjx56hKp8S1IH =uDPN -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Hi Chris, I was monitoring the CPU utilization specifically. I can compromise on 1 less transactions/sec,but 80-90% utilization is not good. I tested the Async Servlet and the Hello World Servlet with the same environment and the client. Hello world servlet gave 20k req/sec and on the other hand Async Servlet was only 100 req/sec even less during the course. Both the results with 100 threads. I have configured 1000 max threads in the connector,and I am testing with 100 concurrent threads. Well I tested a sample html page with Tomcat 6.0.18 and 7.0.30 on Solaris x86 server. The req/sec were almost the same for both,but CPU utilization for Tomcat 6 was 23-27% and for Tomcat 7 it was 80-90% consistently. On the other hand on a Linux system,the req/sec were a little higher in Tomcat 7 say 65k to 55k on tomcat 6 and the CPU utilization was the same for both 6 and 7. My Jprofiler stack trace on Solaris is a lot different. As far as I have observed,for Tomcat 7 the stack Trace leads me to ResponseFacade.setContentType,which was not the behaviour in Tomcat 6. Can that be a bottleneck? Or is there something platform dependent that I should take care of? Thanks. From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, 21 May 2013 9:08 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chirag, On 5/20/13 10:38 AM, Chirag Dewan wrote: I ran my test client on Hello World example servlet on Tomcat 7.0.30. It was 12K req/sec with 80% CPU utilization. The same test case on tomcat 6.0.18 gave me similar req/sec but CPU utilization of 70% . The results on my linux server,with client and server running on the same machine. But all I am doing is comparing the 2 instances in the same environement. Running both client and server on the same machine is not a great test if you want to determine the server load by looking at CPU utilization. Were you looking at overall CPU utilization, or just utilization of the Tomcat JVM process? I am pretty sure the utilization is due to the server. The Jprofiler shows a lot of logging threads in tomcat consuming a lot of memory. Something I should be careful about? Can you be more specific? AFAIK, there are no logging threads in Tomcat at all. Also I ran an asynchronous servlet with the same client. The trans/sec reduced drastically,but the CPU utilization was only 4% I would expect a small reduction in trans/sec when using async, but nothing I would describe as drastic. Can you give us some numbers? Are there too may threads waiting for connection? Would thread pooling help, if we can? I think you mean too many connections waiting for a thread instead of the other way around. Threads are already pooled. What are your Connector configuration details, and what are the parameters for your load test? If you only have 10 connections configured, launching 10,000 clients isn't exactly a fair test. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRm5UNAAoJEBzwKT+lPKRYKoEQAJeWY07c4jr1prAqvdrOyR4X X9CaWrZ4DwvsKuSn2NtZgXqf5lcLhsDzgP3Ej6cNhBx7WcK6zy8zpTZ2RdVo+YOB 1fGfdpZ6rMODT3i2UI8mAQmbG54NwA4C40ShsTpYVtdXb4GfAX+81j8HqtClr5Tf NEBq2xu4LJDc0ajE4mh+Vyr3M3wUG5CEsQAKhj3Owddtus6sltbrBs8116ZzcpQt 10COnt/e08ikyUVuAYxGZPvQ+rwhlFWuvTf0NlG9/oZSDxSb19KcdZ+vMi2HTNQr McQpVxd2RtqpTgM2NJ3uTS6rQ7R4I1moP96DOgNrU6jS2u5sJVoH98weq4AU4uxa 5sXJG4N4eaVsaFE9m2Nio3zgXZDObmF2jh1df79d7TzHkcIJHjtbZ9VHf1gLVqoW v2pWVU+7di6MiDg7P1Y8y2pXwYYF13ZVx6uozyfv6hGuGXDZkMmmg0QatpKguVXl sO3idan5klic3RJJg2zduq6PUlJZyqJJulqJas1DoUHARo6pSDkZWy1hYZQC9BGT dHgmxrA4Jo5Npq1VpiNMCXl/wCT2vKdsseqsCEQN4EIvgJWuZ5l8NkZtceQTPXtE 4CEcsSd83hIDmichkhVw3PoNHKSSFGwdRRI5sEXqW1Vv88Vj7ldtD8518vJUoQLu s2xIFlQjkJKW/W46Y+on =2P5W -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
From: Chirag Dewan [mailto:chirag.dewa...@yahoo.in] Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 Well I tested a sample html page with Tomcat 6.0.18 and 7.0.30 on Solaris x86 server. The req/sec were almost the same for both, but CPU utilization for Tomcat 6 was 23-27% and for Tomcat 7 it was 80-90% consistently. With the exact same JVM for both Tomcat 6 and 7? (32- versus 64-bit, client versus server mode?) On the other hand on a Linux system,the req/sec were a little higher in Tomcat 7 say 65k to 55k on tomcat 6 and the CPU utilization was the same for both 6 and 7. With same JVM as on the Solaris box? (Again, 32- versus 64-bit, client versus server mode?) On the face of it, it sounds like a problem in the Solaris JVM. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RE: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Yes. With same JVM on both Solaris and Linux i.e Java 1.6.39. It is 64 bit version. Thanks. Sent from Yahoo! Mail on Android
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Hi, I have profiled the application using JProfiler,and it seems to me that its my servlet which is taking the majority of time. Though the time is in mili seconds,but I guess since the servlet is code is same as with Tomcat 6.0.18 is it the Servlet 3.0 API which is causing the reduced throughput? Thanks. From: Chirag Dewan chirag.dewa...@yahoo.in To: users@tomcat.apache.org users@tomcat.apache.org; ma...@apache.org ma...@apache.org Sent: Friday, 17 May 2013 6:30 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 I am running it on Solaris x86 server. I haven't even changed my code much. Let me add a profiler,will post the results. Thanks. Sent from Yahoo! Mail on Android
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 20/05/2013 06:59, Chirag Dewan wrote: Hi, I have profiled the application using JProfiler,and it seems to me that its my servlet which is taking the majority of time. Though the time is in mili seconds,but I guess since the servlet is code is same as with Tomcat 6.0.18 is it the Servlet 3.0 API which is causing the reduced throughput? If the time is being spent in your servlet then it can't possibly be the Servlet 3.0 implementation causing the delay since that would be time spent inside Tomcat code, not your code. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
I ran my test client on Hello World example servlet on Tomcat 7.0.30. It was 12K req/sec with 80% CPU utilization. The same test case on tomcat 6.0.18 gave me similar req/sec but CPU utilization of 70% . The results on my linux server,with client and server running on the same machine. But all I am doing is comparing the 2 instances in the same environement. I am pretty sure the utilization is due to the server. The Jprofiler shows a lot of logging threads in tomcat consuming a lot of memory. Something I should be careful about? Also I ran an asynchronous servlet with the same client. The trans/sec reduced drastically,but the CPU utilization was only 4% Are there too may threads waiting for connection? Would thread pooling help,if we can? Thanks. From: Mark Thomas ma...@apache.org To: Tomcat Users List users@tomcat.apache.org Sent: Monday, 20 May 2013 1:48 PM Subject: Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7 On 20/05/2013 06:59, Chirag Dewan wrote: Hi, I have profiled the application using JProfiler,and it seems to me that its my servlet which is taking the majority of time. Though the time is in mili seconds,but I guess since the servlet is code is same as with Tomcat 6.0.18 is it the Servlet 3.0 API which is causing the reduced throughput? If the time is being spent in your servlet then it can't possibly be the Servlet 3.0 implementation causing the delay since that would be time spent inside Tomcat code, not your code. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Hi All, I have upgraded my application from Embedded Tomcat 6 to Embedded Tomcat 7. With Tomcat 6, I was getting around 7 Transactions per sec with a simple HTTP service,which sets the HTTPResponse and returns it. Now with Tomcat 7,I am getting 54000 transactions per sec. Going through the web I figured out that it may be due to Servlet 3.0 API. I have updated my web.xml so that Servlet API doesn't scan the classes for annotations. This is how my web.xml looks like: web-app metadata-complete=true xmlns=http://java.sun.com/xml/ns/javaee; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd; version=3.0 Can anyone provide some other measures I can implement? I am using an HTTP Client to test my performnace. Thanks
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 17/05/2013 11:51, Chirag Dewan wrote: Hi All, I have upgraded my application from Embedded Tomcat 6 to Embedded Tomcat 7. With Tomcat 6, I was getting around 7 Transactions per sec with a simple HTTP service,which sets the HTTPResponse and returns it. Now with Tomcat 7,I am getting 54000 transactions per sec. Going through the web I figured out that it may be due to Servlet 3.0 API. I have updated my web.xml so that Servlet API doesn't scan the classes for annotations. This is how my web.xml looks like: Annotation scanning affects start time, not runtime performance. Can anyone provide some other measures I can implement? Which Tomcat versions are you comparing? Is everything else the same (OS, JVM, etc.)? With trunk I get ~37k req/s with the default configuration and using ab as the client. Just changing maxKeepAliveRequests to -1 increases that to ~43k reqs/sec and all of that without shutting down the stuff I normally have running. There have been changes to the connector code that could add a few microseconds here and there to a request. That could explain the numbers you are seeing. Using a profiler would be the next step to see where the time is being spent. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
I am comparing tomcat 6.0.18 with tomcat 7.0.30. Would adding NIO or APR connector help? Currently I am using default(BIO) connector. Thanks. Sent from Yahoo! Mail on Android
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
Another important factor is the CPU utilization. Earlier for same trans/sec it was 40% now its close to 80%. And yes,the OS,JVM and memory setting are unchanged. Thanks. Sent from Yahoo! Mail on Android
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
On 17/05/2013 13:29, Chirag Dewan wrote: Another important factor is the CPU utilization. Earlier for same trans/sec it was 40% now its close to 80%. I'd upgrade to the latest 7.0.39 but otherwise, definitely time for a profiler. If you want raw performance on kept-alive connections then BIO is the best choice. Just for reference and so I can compare my own tests what hardware and OS are you running this on? And yes,the OS,JVM and memory setting are unchanged. OK. That bit is good. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7
I am running it on Solaris x86 server. I haven't even changed my code much. Let me add a profiler,will post the results. Thanks. Sent from Yahoo! Mail on Android
