Re: Slowness of 4.1
Hi Oleg, Here is some numbers when running TestHttpClient4. I am running from Eclipse using jdk 1. 6.0.22 and a Aspire 5670 which has Intel Core Duo T2300 (1.66 GHz, 667 MHz FSB, 2MB L2 cache) and 2GB of DDR2 memory. Note how for a single thread running 10 requests the response time starts out very slow at 150msec then gets to 1.43msec at the end. httpClient execute time in nanoseconds: 150181377 httpClient execute time in nanoseconds: 2180445 httpClient execute time in nanoseconds: 1540419 httpClient execute time in nanoseconds: 1757207 httpClient execute time in nanoseconds: 1759721 httpClient execute time in nanoseconds: 1492368 httpClient execute time in nanoseconds: 1493765 httpClient execute time in nanoseconds: 1474489 httpClient execute time in nanoseconds: 1456610 httpClient execute time in nanoseconds: 1437613 Document URI: xxx Document Length: 0 bytes Concurrency level: 1 Time taken for tests: 0.187 seconds Complete requests: 10 Failed requests: 0 Content transferred: 0 bytes Requests per second: 53.475933 [#/sec] (mean) Time per request: 18.7 [ms] (mean) When I increase the number of threads to 50 and requests to 50 I get a max req of 2.2K per second and around 400microseconds. The CPU utilization is between 50-60% on the laptop and around 5-7% on my 6 core server with Tomcat 7. Tommow I will setup the test to run from a faster client and see the results. - Original Message From: Tony Anecito To: HttpClient User Discussion Sent: Thu, March 3, 2011 1:06:41 PM Subject: Re: Slowness of 4.1 Thanks Oleg for your explanation about comparitive vs real life. Most of the testing I do is for "real life" since the end goal is user experience thus my comments. I can understand what Apache goals are better now and adjust my expectations accordingly. I did find what I needed as far as examples and hope to get back to response time numbers I have seen with other client side libraries and configurations. I will look at the links you provided to better understand what Apache hopes to do. The only recommendation for stats I would make is have a time measurement for how long each thread takes to execute a request. And show three new stats. One for average time based on that measurement, a 90% of measurements measurement (Gives indication of distribution) and average response time over load for each average. Just some thoughts you and the team can decide how if at all those suggestions might fit your testing goals. Regards, -Tony - Original Message From: Oleg Kalnichevski To: HttpClient User Discussion <mailto:httpclient-users@hc.apache.org> Sent: Thu, March 3, 2011 1:52:37 PM Subject: Re: Slowness of 4.1 On Thu, 2011-03-03 at 09:20 -0800, Tony Anecito wrote: > Hi, > > Where can I find the client side code for this test? I think I found it once > before and discovered the way the response times were calculated were >incorrect. > What I saw indicated your test results were divideing the number of requests > into the time it took for all of them to complete when the tests were run in > parallel so it made it look like the requests were faster than they really >were. > > It was like taking ten 100ms requests run in parallel and getting 10ms per > request when in reality it was 100ms. > I did see your remarks about the benchmark on the tomcat user list. (1) the benchmark tries to closely simulate Apache Bench (ab), which is a well established and widely used tool for performance measurement. http://en.wikipedia.org/wiki/ApacheBench (2) the benchmark is not intended to calculate 'real-life' or 'accurate' response time, whatever that means. The purpose of the benchmark is to provide a _baseline_ for a comparative performance analysis and is intended to give a _rough_ indication as to whether an HTTP client A faster than an client HTTP, or whether data throughput increased or decreased after particular set of changes. That is it. Having said all that you are very welcome to suggest improvements to the benchmark or a more accurate algorithm for calculating performance numbers. Oleg > In either case I will see what if I can find again the client side code > soemwhere and compare it to what I have unless you have a link to it. > > Thanks, > -Tony > > > > - Original Message > From: Oleg Kalnichevski > To: HttpClient User Discussion <mailto:httpclient-users@hc.apache.org> > Sent: Thu, March 3, 2011 2:46:03 AM > Subject: Re: Slowness of 4.1 > > On Thu, 2011-03-03 at 01:38 -0800, Tony Anecito wrote: > > Hi All, > > > > I downloaded httpclient 4.1 and noticed it is significantly slower than > > 3.1. >I > > > > even used the threadsafe connection manager hoping for better performance. > > I > > used to get below 3msec and now it is above 150msec.
Re: Slowness of 4.1
Thanks Oleg for your explanation about comparitive vs real life. Most of the testing I do is for "real life" since the end goal is user experience thus my comments. I can understand what Apache goals are better now and adjust my expectations accordingly. I did find what I needed as far as examples and hope to get back to response time numbers I have seen with other client side libraries and configurations. I will look at the links you provided to better understand what Apache hopes to do. The only recommendation for stats I would make is have a time measurement for how long each thread takes to execute a request. And show three new stats. One for average time based on that measurement, a 90% of measurements measurement (Gives indication of distribution) and average response time over load for each average. Just some thoughts you and the team can decide how if at all those suggestions might fit your testing goals. Regards, -Tony - Original Message From: Oleg Kalnichevski To: HttpClient User Discussion Sent: Thu, March 3, 2011 1:52:37 PM Subject: Re: Slowness of 4.1 On Thu, 2011-03-03 at 09:20 -0800, Tony Anecito wrote: > Hi, > > Where can I find the client side code for this test? I think I found it once > before and discovered the way the response times were calculated were >incorrect. > What I saw indicated your test results were divideing the number of requests > into the time it took for all of them to complete when the tests were run in > parallel so it made it look like the requests were faster than they really >were. > > It was like taking ten 100ms requests run in parallel and getting 10ms per > request when in reality it was 100ms. > I did see your remarks about the benchmark on the tomcat user list. (1) the benchmark tries to closely simulate Apache Bench (ab), which is a well established and widely used tool for performance measurement. http://en.wikipedia.org/wiki/ApacheBench (2) the benchmark is not intended to calculate 'real-life' or 'accurate' response time, whatever that means. The purpose of the benchmark is to provide a _baseline_ for a comparative performance analysis and is intended to give a _rough_ indication as to whether an HTTP client A faster than an client HTTP, or whether data throughput increased or decreased after particular set of changes. That is it. Having said all that you are very welcome to suggest improvements to the benchmark or a more accurate algorithm for calculating performance numbers. Oleg > In either case I will see what if I can find again the client side code > soemwhere and compare it to what I have unless you have a link to it. > > Thanks, > -Tony > > > > - Original Message > From: Oleg Kalnichevski > To: HttpClient User Discussion <mailto:httpclient-users@hc.apache.org> > Sent: Thu, March 3, 2011 2:46:03 AM > Subject: Re: Slowness of 4.1 > > On Thu, 2011-03-03 at 01:38 -0800, Tony Anecito wrote: > > Hi All, > > > > I downloaded httpclient 4.1 and noticed it is significantly slower than > > 3.1. >I > > > > even used the threadsafe connection manager hoping for better performance. > > I > > used to get below 3msec and now it is above 150msec. > > Is keepalive used by default? If not where can I find an example that sets > > it > > > using either a scheme, connection manager httpclient instance or the > > httpget > > object? > > > > What kind of response time should I expect hitting a servlet with no code > > in > >the > > > > method on a 2.8Ghz 6 core AMD server? > > > > Any ideas on what it might be? I used code sample from the site and it > > works > > just very slow. > > > > Thanks, > > -Tony > > > > HttpClient 4.1 is known to be comfortably faster than 3.1. There is > likely to be a problem with how you are measuring performance / response > time. > > http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore > > Oleg > > > > - > To unsubscribe, e-mail: >mailto:mailto:httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: >mailto:mailto:httpclient-users-h...@hc.apache.org > > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Slowness of 4.1
On Thu, 2011-03-03 at 09:20 -0800, Tony Anecito wrote: > Hi, > > Where can I find the client side code for this test? I think I found it once > before and discovered the way the response times were calculated were > incorrect. > What I saw indicated your test results were divideing the number of requests > into the time it took for all of them to complete when the tests were run in > parallel so it made it look like the requests were faster than they really > were. > It was like taking ten 100ms requests run in parallel and getting 10ms per > request when in reality it was 100ms. > I did see your remarks about the benchmark on the tomcat user list. (1) the benchmark tries to closely simulate Apache Bench (ab), which is a well established and widely used tool for performance measurement. http://en.wikipedia.org/wiki/ApacheBench (2) the benchmark is not intended to calculate 'real-life' or 'accurate' response time, whatever that means. The purpose of the benchmark is to provide a _baseline_ for a comparative performance analysis and is intended to give a _rough_ indication as to whether an HTTP client A faster than an client HTTP, or whether data throughput increased or decreased after particular set of changes. That is it. Having said all that you are very welcome to suggest improvements to the benchmark or a more accurate algorithm for calculating performance numbers. Oleg > In either case I will see what if I can find again the client side code > soemwhere and compare it to what I have unless you have a link to it. > > Thanks, > -Tony > > > > - Original Message > From: Oleg Kalnichevski > To: HttpClient User Discussion > Sent: Thu, March 3, 2011 2:46:03 AM > Subject: Re: Slowness of 4.1 > > On Thu, 2011-03-03 at 01:38 -0800, Tony Anecito wrote: > > Hi All, > > > > I downloaded httpclient 4.1 and noticed it is significantly slower than > > 3.1. I > > > even used the threadsafe connection manager hoping for better performance. > > I > > used to get below 3msec and now it is above 150msec. > > Is keepalive used by default? If not where can I find an example that sets > > it > > using either a scheme, connection manager httpclient instance or the > > httpget > > object? > > > > What kind of response time should I expect hitting a servlet with no code > > in > >the > > > > method on a 2.8Ghz 6 core AMD server? > > > > Any ideas on what it might be? I used code sample from the site and it > > works > > just very slow. > > > > Thanks, > > -Tony > > > > HttpClient 4.1 is known to be comfortably faster than 3.1. There is > likely to be a problem with how you are measuring performance / response > time. > > http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore > > Oleg > > > > - > To unsubscribe, e-mail: mailto:httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: mailto:httpclient-users-h...@hc.apache.org > > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Slowness of 4.1
Hi Oleg, I think I found the test client code for 4.x it is: http://svn.apache.org/repos/asf/httpcomponents/httpclient/trunk/httpclient-benchmark/src/main/java/org/apache/http/client/benchmark/TestHttpClient4.java Thanks for the help I will let you know how things turn out. -Tony - Original Message From: Tony Anecito To: HttpClient User Discussion Sent: Thu, March 3, 2011 10:20:18 AM Subject: Re: Slowness of 4.1 Hi, Where can I find the client side code for this test? I think I found it once before and discovered the way the response times were calculated were incorrect. What I saw indicated your test results were divideing the number of requests into the time it took for all of them to complete when the tests were run in parallel so it made it look like the requests were faster than they really were. It was like taking ten 100ms requests run in parallel and getting 10ms per request when in reality it was 100ms. In either case I will see what if I can find again the client side code soemwhere and compare it to what I have unless you have a link to it. Thanks, -Tony - Original Message From: Oleg Kalnichevski To: HttpClient User Discussion <mailto:httpclient-users@hc.apache.org> Sent: Thu, March 3, 2011 2:46:03 AM Subject: Re: Slowness of 4.1 On Thu, 2011-03-03 at 01:38 -0800, Tony Anecito wrote: > Hi All, > > I downloaded httpclient 4.1 and noticed it is significantly slower than 3.1. > I > even used the threadsafe connection manager hoping for better performance. I > used to get below 3msec and now it is above 150msec. > Is keepalive used by default? If not where can I find an example that sets it > using either a scheme, connection manager httpclient instance or the httpget > object? > > What kind of response time should I expect hitting a servlet with no code in >the > > method on a 2.8Ghz 6 core AMD server? > > Any ideas on what it might be? I used code sample from the site and it works > just very slow. > > Thanks, > -Tony > HttpClient 4.1 is known to be comfortably faster than 3.1. There is likely to be a problem with how you are measuring performance / response time. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore Oleg - To unsubscribe, e-mail: mailto:mailto:httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: mailto:mailto:httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Slowness of 4.1
Hi, Where can I find the client side code for this test? I think I found it once before and discovered the way the response times were calculated were incorrect. What I saw indicated your test results were divideing the number of requests into the time it took for all of them to complete when the tests were run in parallel so it made it look like the requests were faster than they really were. It was like taking ten 100ms requests run in parallel and getting 10ms per request when in reality it was 100ms. In either case I will see what if I can find again the client side code soemwhere and compare it to what I have unless you have a link to it. Thanks, -Tony - Original Message From: Oleg Kalnichevski To: HttpClient User Discussion Sent: Thu, March 3, 2011 2:46:03 AM Subject: Re: Slowness of 4.1 On Thu, 2011-03-03 at 01:38 -0800, Tony Anecito wrote: > Hi All, > > I downloaded httpclient 4.1 and noticed it is significantly slower than 3.1. > I > even used the threadsafe connection manager hoping for better performance. I > used to get below 3msec and now it is above 150msec. > Is keepalive used by default? If not where can I find an example that sets it > using either a scheme, connection manager httpclient instance or the httpget > object? > > What kind of response time should I expect hitting a servlet with no code in >the > > method on a 2.8Ghz 6 core AMD server? > > Any ideas on what it might be? I used code sample from the site and it works > just very slow. > > Thanks, > -Tony > HttpClient 4.1 is known to be comfortably faster than 3.1. There is likely to be a problem with how you are measuring performance / response time. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore Oleg - To unsubscribe, e-mail: mailto:httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: mailto:httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Slowness of 4.1
On Thu, 2011-03-03 at 01:38 -0800, Tony Anecito wrote: > Hi All, > > I downloaded httpclient 4.1 and noticed it is significantly slower than 3.1. > I > even used the threadsafe connection manager hoping for better performance. I > used to get below 3msec and now it is above 150msec. > Is keepalive used by default? If not where can I find an example that sets it > using either a scheme, connection manager httpclient instance or the httpget > object? > > What kind of response time should I expect hitting a servlet with no code in > the > method on a 2.8Ghz 6 core AMD server? > > Any ideas on what it might be? I used code sample from the site and it works > just very slow. > > Thanks, > -Tony > HttpClient 4.1 is known to be comfortably faster than 3.1. There is likely to be a problem with how you are measuring performance / response time. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Slowness of 4.1
Hi All, I downloaded httpclient 4.1 and noticed it is significantly slower than 3.1. I even used the threadsafe connection manager hoping for better performance. I used to get below 3msec and now it is above 150msec. Is keepalive used by default? If not where can I find an example that sets it using either a scheme, connection manager httpclient instance or the httpget object? What kind of response time should I expect hitting a servlet with no code in the method on a 2.8Ghz 6 core AMD server? Any ideas on what it might be? I used code sample from the site and it works just very slow. Thanks, -Tony - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
