Re: Thread Contention (due to use of proxys) in HttpClient 4.3
>> 2) Can I expect 4.2 to scale better? > > I do not think so. In my tests HC 4.3 performs better than 4.2. There > have also been reports Even with high concurrency of say 200 and high TPS, for example in a http proxy scenario? >> 3) Is there a way of configuring 4.3.3 to not use proxys? > > No, there is not. The cost of dynamic proxies in modern JREs is believed > to be low. The problem I see isn't the use of proxies, it is in the creation of these proxy instances which is synchnorized on a HashMap 'cache' in two places in java.lang.reflect.Proxy.getProxyClass0(ClassLoader, Class...). This is visible in the profiler screenshot I shared too. This probably isn't an issue with low concurrency, or high concurrency and low TPS, but in my testing/profiling it comes up as an issue. thanks, Dan - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Mon, 2014-05-12 at 13:48 +0200, Oleg Kalnichevski wrote: > On Mon, 2014-05-12 at 00:13 +0100, Daniel Feist wrote: > > Hi, > > > > I'm using HttpClient in a situation where high concurrency is expected > > and am doing some testing/benchmarking using gatling-tool. > > > > While performance isn't bad, things aren't scaling as well as I'd have > > hoped. (I'm running on a amazon XL instance) > > > > When profiling with gatling running using 200 users I see very > > significant contention in java.reflect.Proxy.newInstance(). See: > > https://www.dropbox.com/s/jjaqyfm54lxw0s8/Screen%20Shot%202014-05-08%20at%201.34.11%20PM.png > > > > When i look at 4.2 source code, Proxys aren't used in this two places. > > > > Questions: > > 1) Is this known issue? > > No, it is not > > > 2) Can I expect 4.2 to scale better? > > I do not think so. In my tests HC 4.3 performs better than 4.2. There > have also been > independent reports on the user list confirming that. Sorry. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Mon, 2014-05-12 at 13:21 +0100, Daniel Feist wrote: > >> 2) Can I expect 4.2 to scale better? > > > > I do not think so. In my tests HC 4.3 performs better than 4.2. There > > have also been reports > > Even with high concurrency of say 200 and high TPS, for example in a > http proxy scenario? > Yes, also with over 200 concurrent connections. However, for HTTP proxy scenarios one may be advised to consider using a non-blocking model. > >> 3) Is there a way of configuring 4.3.3 to not use proxys? > > > > No, there is not. The cost of dynamic proxies in modern JREs is believed > > to be low. > > The problem I see isn't the use of proxies, it is in the creation of > these proxy instances which is synchnorized on a HashMap 'cache' in > two places in java.lang.reflect.Proxy.getProxyClass0(ClassLoader, > Class...). This is visible in the profiler screenshot I shared > too. > > This probably isn't an issue with low concurrency, or high concurrency > and low TPS, but in my testing/profiling it comes up as an issue. > I use this micro-benchmark to benchmark HttpClient performance between versions as well compared to other implementations. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore I am open to the idea of replacing dynamic proxies with hand crafted classes, but I would like to see more evidence that java.lang.reflect.Proxy is indeed a bottleneck. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Mon, 2014-05-12 at 00:13 +0100, Daniel Feist wrote: > Hi, > > I'm using HttpClient in a situation where high concurrency is expected > and am doing some testing/benchmarking using gatling-tool. > > While performance isn't bad, things aren't scaling as well as I'd have > hoped. (I'm running on a amazon XL instance) > > When profiling with gatling running using 200 users I see very > significant contention in java.reflect.Proxy.newInstance(). See: > https://www.dropbox.com/s/jjaqyfm54lxw0s8/Screen%20Shot%202014-05-08%20at%201.34.11%20PM.png > > When i look at 4.2 source code, Proxys aren't used in this two places. > > Questions: > 1) Is this known issue? No, it is not > 2) Can I expect 4.2 to scale better? I do not think so. In my tests HC 4.3 performs better than 4.2. There have also been reports > 3) Is there a way of configuring 4.3.3 to not use proxys? No, there is not. The cost of dynamic proxies in modern JREs is believed to be low. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
I have seen similar issues sometimes - while running with 512 concurrent connections using HttpClient 4.3 (Noticed all threads stuck in java.lang.reflect.Proxy.getProxyClass0 in jstack log). I found some discussion online regarding improving the performance in latest jre but have not tried running my application with jdk8 yet. http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-April/subject.html#15854 Jaikit On Monday, May 12, 2014 11:17 AM, Oleg Kalnichevski wrote: On Mon, 2014-05-12 at 13:21 +0100, Daniel Feist wrote: > >> 2) Can I expect 4.2 to scale better? > > > > I do not think so. In my tests HC 4.3 performs better than 4.2. There > > have also been reports > > Even with high concurrency of say 200 and high TPS, for example in a > http proxy scenario? > Yes, also with over 200 concurrent connections. However, for HTTP proxy scenarios one may be advised to consider using a non-blocking model. > >> 3) Is there a way of configuring 4.3.3 to not use proxys? > > > > No, there is not. The cost of dynamic proxies in modern JREs is believed > > to be low. > > The problem I see isn't the use of proxies, it is in the creation of > these proxy instances which is synchnorized on a HashMap 'cache' in > two places in java.lang.reflect.Proxy.getProxyClass0(ClassLoader, > Class...). This is visible in the profiler screenshot I shared > too. > > This probably isn't an issue with low concurrency, or high concurrency > and low TPS, but in my testing/profiling it comes up as an issue. > I use this micro-benchmark to benchmark HttpClient performance between versions as well compared to other implementations. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore I am open to the idea of replacing dynamic proxies with hand crafted classes, but I would like to see more evidence that java.lang.reflect.Proxy is indeed a bottleneck. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
Jaikit, Thats exactly what I was seeing, also with Java7. Let me do some testing though, because just because you see contention in profiler, doesn't mean that there is also a significant impact. I'll report back when I've done this. Dan On Mon, May 12, 2014 at 10:10 PM, Jaikit Savla wrote: > I have seen similar issues sometimes - while running with 512 concurrent > connections using HttpClient 4.3 (Noticed all threads stuck in > java.lang.reflect.Proxy.getProxyClass0 in jstack log). I found some > discussion online regarding improving the performance in latest jre but have > not tried running my application with jdk8 yet. > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-April/subject.html#15854 > > Jaikit > > On Monday, May 12, 2014 11:17 AM, Oleg Kalnichevski wrote: > > On Mon, 2014-05-12 at 13:21 +0100, Daniel Feist wrote: >> >> 2) Can I expect 4.2 to scale better? >> > >> > I do not think so. In my tests HC 4.3 performs better than 4.2. There >> > have also been reports >> >> Even with high concurrency of say 200 and high TPS, for example in a >> http proxy scenario? >> > > Yes, also with over 200 concurrent connections. However, for HTTP proxy > scenarios one may be advised to consider using a non-blocking model. > >> >> 3) Is there a way of configuring 4.3.3 to not use proxys? >> > >> > No, there is not. The cost of dynamic proxies in modern JREs is believed >> > to be low. >> >> The problem I see isn't the use of proxies, it is in the creation of >> these proxy instances which is synchnorized on a HashMap 'cache' in >> two places in java.lang.reflect.Proxy.getProxyClass0(ClassLoader, >> Class...). This is visible in the profiler screenshot I shared >> too. >> >> This probably isn't an issue with low concurrency, or high concurrency >> and low TPS, but in my testing/profiling it comes up as an issue. >> > > I use this micro-benchmark to benchmark HttpClient performance between > versions as well compared to other implementations. > > http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore > > I am open to the idea of replacing dynamic proxies with hand crafted > classes, but I would like to see more evidence that > java.lang.reflect.Proxy is indeed a bottleneck. > > Oleg > > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Mon, 2014-05-12 at 14:10 -0700, Jaikit Savla wrote: > I have seen similar issues sometimes - while running with 512 concurrent > connections using HttpClient 4.3 (Noticed all threads stuck in > java.lang.reflect.Proxy.getProxyClass0 in jstack log). I found some > discussion online regarding improving the performance in latest jre but have > not tried running my application with jdk8 yet. > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-April/subject.html#15854 > > Jaikit > Jaikit & Daniel, Dynamic proxies help eliminate a great deal of unnecessary code but I have no problem with replacing them with hand crafted classes in order to reduce lock contention. Feel free to raise an enhancement request and contribute patches. Oleg > On Monday, May 12, 2014 11:17 AM, Oleg Kalnichevski wrote: > > On Mon, 2014-05-12 at 13:21 +0100, Daniel Feist wrote: > > >> 2) Can I expect 4.2 to scale better? > > > > > > I do not think so. In my tests HC 4.3 performs better than 4.2. There > > > have also been reports > > > > Even with high concurrency of say 200 and high TPS, for example in a > > http proxy scenario? > > > > Yes, also with over 200 concurrent connections. However, for HTTP proxy > scenarios one may be advised to consider using a non-blocking model. > > > >> 3) Is there a way of configuring 4.3.3 to not use proxys? > > > > > > No, there is not. The cost of dynamic proxies in modern JREs is believed > > > to be low. > > > > The problem I see isn't the use of proxies, it is in the creation of > > these proxy instances which is synchnorized on a HashMap 'cache' in > > two places in java.lang.reflect.Proxy.getProxyClass0(ClassLoader, > > Class...). This is visible in the profiler screenshot I shared > > too. > > > > This probably isn't an issue with low concurrency, or high concurrency > > and low TPS, but in my testing/profiling it comes up as an issue. > > > > I use this micro-benchmark to benchmark HttpClient performance between > versions as well compared to other implementations. > > http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore > > I am open to the idea of replacing dynamic proxies with hand crafted > classes, but I would like to see more evidence that > java.lang.reflect.Proxy is indeed a bottleneck. > > Oleg > > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
Hi, I spent a good part of yesterday testing and comparing HttpClient performance and the results are interesting: HttpClient 3.1 is 20% faster than HttpClient 4.3 when both configured in the same way and when using 50 client threads! https://docs.google.com/a/mulesoft.com/spreadsheets/d/1Dqp2dH7K8nQeBzAC3S4hboYmSm67GXWO6AYgTYiVInQ/edit#gid=0 When profiling each version when running with 200 threads, this is what I see: HttpClient4.3: - Significant amount of contention due to creation of proxies as mentioned before. HttpClient4.2 -> - No blocked threads HttpClient3.1: -> - Very significant amount of contention due to synchronized method: org.apache.commons.httpclient.params.DefaultHttpParams.getParameter(String) used multiple times for each request. - Minimal contention getting/returning connections from pool. I think it's clear to see why 4.2 scale a bit better, but I'm unsure why 4.3 (when not using minimal) would perform worse than 4.3.. are there additional features in 4.3? Notes: - - I used your http client benchmark project referenced by Oleg - I ran tests on Amazon XL (32-core) VM, ensuring each test ran for 30s+. - I tweaked the Jetty configuration to make jetty perform slightly better (see below). I verified changes were faster by testing before and after this change. - For each version of HttpClient i configured the same bufferSize, soTimeout and disabled stale connection checking. - Also recorded as the values for HttpClient4.3 minimal client. Be interested to hear you thoughts, and let me know if you'd like any more information. thanks, Dan JettyConfig: final BlockingChannelConnector connector = new BlockingChannelConnector(); final Server server = new Server(); server.addConnector(connector); server.setHandler(new RandomDataHandler()); server.setThreadPool(new org.eclipse.jetty.util.thread.QueuedThreadPool(500)); > Jaikit & Daniel, > > Dynamic proxies help eliminate a great deal of unnecessary code but I > have no problem with replacing them with hand crafted classes in order > to reduce lock contention. > > Feel free to raise an enhancement request and contribute patches. > > Oleg > >> On Monday, May 12, 2014 11:17 AM, Oleg Kalnichevski wrote: >> >> On Mon, 2014-05-12 at 13:21 +0100, Daniel Feist wrote: >> > >> 2) Can I expect 4.2 to scale better? >> > > >> > > I do not think so. In my tests HC 4.3 performs better than 4.2. There >> > > have also been reports >> > >> > Even with high concurrency of say 200 and high TPS, for example in a >> > http proxy scenario? >> > >> >> Yes, also with over 200 concurrent connections. However, for HTTP proxy >> scenarios one may be advised to consider using a non-blocking model. >> >> > >> 3) Is there a way of configuring 4.3.3 to not use proxys? >> > > >> > > No, there is not. The cost of dynamic proxies in modern JREs is believed >> > > to be low. >> > >> > The problem I see isn't the use of proxies, it is in the creation of >> > these proxy instances which is synchnorized on a HashMap 'cache' in >> > two places in java.lang.reflect.Proxy.getProxyClass0(ClassLoader, >> > Class...). This is visible in the profiler screenshot I shared >> > too. >> > >> > This probably isn't an issue with low concurrency, or high concurrency >> > and low TPS, but in my testing/profiling it comes up as an issue. >> > >> >> I use this micro-benchmark to benchmark HttpClient performance between >> versions as well compared to other implementations. >> >> http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore >> >> I am open to the idea of replacing dynamic proxies with hand crafted >> classes, but I would like to see more evidence that >> java.lang.reflect.Proxy is indeed a bottleneck. >> >> Oleg >> >> >> >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
> I do not think they are configured the same way. For one, HC 4.3 uses > content decompression by default whereas HC 3.1 does not. That obviously > skews the results. Maybe but doesn't look like it, I just updated the results using 4.3 and turning off everything: HttpClients.custom().setConnectionManager(this.mgr).disableAuthCaching().disableAutomaticRetries().disableConnectionState().disableContentCompression() .disableCookieManagement().disableRedirectHandling().setDefaultRequestConfig(RequestConfig.custom().setStaleConnectionCheckEnabled(false).build()).build(); 4.3 Improves a bit, but is still slower than 3.1 and 4.2, and I'm pretty sure that with this config those previous version have *more* functionality. I also compared using "new DefaultHttpClient" in 4.3. > I stopped comparing HC 4.x to 3.1 a while ago because in my tests HC 3.1 > was consistently slower. I can dig out that code and re-run the tests. Hmm, I'm not seeing that at all. >> I think it's clear to see why 4.2 scale a bit better, > > If you are reasonably sure that dynamic proxies are the culprit let's > get rid of them and re-run the tests. I'm not convinced at all no, because after sending this email I see that minimalClient scales just as well as 4.2 and minimalClient uses proxies too. Maybe use of proxy is causing some other issue, but it doesn't seems to be affecting scalability as I first assumed it was. > Of course. HC 4.3 in full configuration simply does much more: > transparent content decompression, connection state tracking, auth > caching. You need to disable those features to have a fairer > comparison. I turned all that off though, still slower. >> - I tweaked the Jetty configuration to make jetty perform slightly >> better (see below). I verified changes were faster by testing before >> and after this change. > > Can I incorporate these changes into the project? Sure, but take into account that while this connector gives better results for blocking clients I'm not sure how it affects non-blocking clients. > In my opinion any difference within 10% is simply unsubstantial and can > well be a fluke. Overall, based on these numbers I am happy with HC 4.3 > performance. I will happily make it faster, though. I agree 10% isn't that much and this testing maybe isn't 100% through, but I think it's fairly clear that unless you use minimalClient with 4.3, it doesn't matter which features you turn off (even all of them), 4.3 still doesn't perform as fast as 4.2. Also I'm surprised that 3.1 runs so fast, even if it doesn't scale quite as well as 4.x. Couple of outstanding question: 1) I assume your recommendation about leaving stale connection checking on by default unless you know what you are doing is still the same in 4.3? Maybe it's a good idea to turn if off by default for idempotent http methods and on by default for others, rather than having it on for everything? 2) I'm still unsure why 4.3 is running slower than 4.2, even with everything turned off.. For my particular use case i think I might just stick to 3.1 for now, until our next minor release anyway, because the huge performance difference I was seeing between 3.1 and 4.3 it seems was 100% down to the stale connection check! That said i like the fluent interface and flexibility of 4.3. Dan > > Oleg > >> >> thanks, >> Dan >> > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Wed, 2014-05-14 at 11:46 +0100, Daniel Feist wrote: > Hi, > > I spent a good part of yesterday testing and comparing HttpClient > performance and the results are interesting: HttpClient 3.1 is 20% > faster than HttpClient 4.3 when both configured in the same way and > when using 50 client threads! > Daniel, I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for me) HC 4.3 comfortably outperformed HC 3.1. Theories are welcome. Oleg PS: I'll work on dynamic proxy removal this weekend but you are welcome to the lead if you are so inclined --- oleg@ubuntu:~/src/apache.org/httpcomponents/httpclient-benchmark$ gr -q benchmark = HTTP agent: Apache HttpClient 3.1 - 100 POST requests - Document URI: http://localhost:53796/echo Document Length:2048 bytes Concurrency level: 50 Time taken for tests: 64.625 seconds Complete requests: 100 Failed requests:0 Content transferred:204800 bytes Requests per second:15473.888 [#/sec] (mean) - = HTTP agent: Apache HttpClient (ver: 4.3.3) - 100 POST requests - Document URI: http://localhost:47900/echo Document Length:2048 bytes Concurrency level: 50 Time taken for tests: 55.837 seconds Complete requests: 100 Failed requests:0 Content transferred:204800 bytes Requests per second:17909.271 [#/sec] (mean) - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Wed, 2014-05-14 at 11:46 +0100, Daniel Feist wrote: > Hi, > > I spent a good part of yesterday testing and comparing HttpClient > performance and the results are interesting: HttpClient 3.1 is 20% > faster than HttpClient 4.3 when both configured in the same way and > when using 50 client threads! > I do not think they are configured the same way. For one, HC 4.3 uses content decompression by default whereas HC 3.1 does not. That obviously skews the results. I stopped comparing HC 4.x to 3.1 a while ago because in my tests HC 3.1 was consistently slower. I can dig out that code and re-run the tests. > https://docs.google.com/a/mulesoft.com/spreadsheets/d/1Dqp2dH7K8nQeBzAC3S4hboYmSm67GXWO6AYgTYiVInQ/edit#gid=0 > > When profiling each version when running with 200 threads, this is what I see: > > HttpClient4.3: > - Significant amount of contention due to creation of proxies as > mentioned before. > > HttpClient4.2 -> > - No blocked threads > > HttpClient3.1: -> > - Very significant amount of contention due to synchronized method: > org.apache.commons.httpclient.params.DefaultHttpParams.getParameter(String) > used multiple times for each request. > - Minimal contention getting/returning connections from pool. > > > I think it's clear to see why 4.2 scale a bit better, If you are reasonably sure that dynamic proxies are the culprit let's get rid of them and re-run the tests. > but I'm unsure > why 4.3 (when not using minimal) would perform worse than 4.3.. are > there additional features in 4.3? > Of course. HC 4.3 in full configuration simply does much more: transparent content decompression, connection state tracking, auth caching. You need to disable those features to have a fairer comparison. > > Notes: > - > - I used your http client benchmark project referenced by Oleg > - I ran tests on Amazon XL (32-core) VM, ensuring each test ran for 30s+. > - I tweaked the Jetty configuration to make jetty perform slightly > better (see below). I verified changes were faster by testing before > and after this change. Can I incorporate these changes into the project? > - For each version of HttpClient i configured the same bufferSize, > soTimeout and disabled stale connection checking. > - Also recorded as the values for HttpClient4.3 minimal client. > > > Be interested to hear you thoughts, and let me know if you'd like any > more information. > In my opinion any difference within 10% is simply unsubstantial and can well be a fluke. Overall, based on these numbers I am happy with HC 4.3 performance. I will happily make it faster, though. Oleg > > thanks, > Dan > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
> I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > me) HC 4.3 comfortably outperformed HC 3.1. Yes I see the same running this revision. The difference is you are testing: HttpClient31 vs. HttpClients.createMinimal(mgr) and I was testing HttpClient31 vs. HttpClient.custom().setMgr(mgr).disableEverything(true).build(). For some reason the HttpClient built by HttpClientBuilder, even when everything is turned is not only slower than minimalClient (to be expected) but also slower than DefaultHttpClient in previous versions. Not a major difference, but definitely the inverse of what of whats in the results you sent. I'm still not seeing a major issue with the contention i mentioned i'd seen in profiler, be an interesting experiment to test performance with 500 treads, both with/without proxies though. Dan > oleg@ubuntu:~/src/apache.org/httpcomponents/httpclient-benchmark$ gr -q > benchmark > = > HTTP agent: Apache HttpClient 3.1 > - > 100 POST requests > - > Document URI: http://localhost:53796/echo > Document Length:2048 bytes > > Concurrency level: 50 > Time taken for tests: 64.625 seconds > Complete requests: 100 > Failed requests:0 > Content transferred:204800 bytes > Requests per second:15473.888 [#/sec] (mean) > - > = > HTTP agent: Apache HttpClient (ver: 4.3.3) > - > 100 POST requests > - > Document URI: http://localhost:47900/echo > Document Length:2048 bytes > > Concurrency level: 50 > Time taken for tests: 55.837 seconds > Complete requests: 100 > Failed requests:0 > Content transferred:204800 bytes > Requests per second:17909.271 [#/sec] (mean) > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
> I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > me) HC 4.3 comfortably outperformed HC 3.1. > > Theories are welcome. Really don't know. Current theory is that with 32 core machine contention somehow isn't an issue. Currently re-running on dual-core machine, I'll update spreadsheets with results in a separate tab. > PS: I'll work on dynamic proxy removal this weekend but you are welcome > to the lead if you are so inclined Not sure I'd rush to do this, as so far there isn't an proof that it's an issue. Something is making 4.3 slower than 4.2 when not using minimal client (even with everything turned off), but it not the proxies as those are used with minimal client too. Let's re-asses with results from 2-core machine.. Dan > > --- > oleg@ubuntu:~/src/apache.org/httpcomponents/httpclient-benchmark$ gr -q > benchmark > = > HTTP agent: Apache HttpClient 3.1 > - > 100 POST requests > - > Document URI: http://localhost:53796/echo > Document Length:2048 bytes > > Concurrency level: 50 > Time taken for tests: 64.625 seconds > Complete requests: 100 > Failed requests:0 > Content transferred:204800 bytes > Requests per second:15473.888 [#/sec] (mean) > - > = > HTTP agent: Apache HttpClient (ver: 4.3.3) > - > 100 POST requests > - > Document URI: http://localhost:47900/echo > Document Length:2048 bytes > > Concurrency level: 50 > Time taken for tests: 55.837 seconds > Complete requests: 100 > Failed requests:0 > Content transferred:204800 bytes > Requests per second:17909.271 [#/sec] (mean) > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Thu, 2014-05-15 at 01:10 +0100, Daniel Feist wrote: > > I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > > me) HC 4.3 comfortably outperformed HC 3.1. > > Yes I see the same running this revision. The difference is you are testing: > > HttpClient31 vs. HttpClients.createMinimal(mgr) > > and I was testing > > HttpClient31 vs. > HttpClient.custom().setMgr(mgr).disableEverything(true).build(). > > For some reason the HttpClient built by HttpClientBuilder, even when > everything is turned is not only slower than minimalClient (to be > expected) but also slower than DefaultHttpClient in previous versions. > Not a major difference, but definitely the inverse of what of whats > in the results you sent. > I am sorry I cannot reproduce it. I replaced HC in minimal configuration with this --- final RequestConfig requestConfig = RequestConfig.custom() .setStaleConnectionCheckEnabled(false) .setExpectContinueEnabled(false) .build(); this.httpclient = HttpClients.custom() .setConnectionManager(this.mgr) .disableContentCompression() .setDefaultRequestConfig(requestConfig) .build(); --- and still had HC 4.3 outperforming HC 3.1 Oleg --- oleg@ubuntu:~/src/apache.org/httpcomponents/httpclient-benchmark$ gr benchmark :compileJava UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :benchmark = HTTP agent: Apache HttpClient 3.1 - 100 POST requests - Document URI: http://localhost:35116/echo Document Length:2048 bytes Concurrency level: 50 Time taken for tests: 58.16 seconds Complete requests: 100 Failed requests:0 Content transferred:204800 bytes Requests per second:17193.947 [#/sec] (mean) - = HTTP agent: Apache HttpClient (ver: 4.3.3) - 100 POST requests - Document URI: http://localhost:53174/echo Document Length:2048 bytes Concurrency level: 50 Time taken for tests: 53.31 seconds Complete requests: 100 Failed requests:0 Content transferred:204800 bytes Requests per second:18758.207 [#/sec] (mean) - BUILD SUCCESSFUL - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Thu, 2014-05-15 at 01:10 +0100, Daniel Feist wrote: > > I re-ran the benchmark (r1594594) on my computer and unsurprisingly (for > > me) HC 4.3 comfortably outperformed HC 3.1. > > Yes I see the same running this revision. Could you please find out if this difference is consistent regardless of the CPU core number used by the system? The only viable theory that I have at the moment is for some reason HC 4.3 does not scale well on 4+ CPU core system. I have only very moderate hardware at my disposal. I have never had a chance to test HC performance on a big box with lots of CPUs and therefore may have missed something. > The difference is you are testing: > > HttpClient31 vs. HttpClients.createMinimal(mgr) > > and I was testing > > HttpClient31 vs. > HttpClient.custom().setMgr(mgr).disableEverything(true).build(). > > > For some reason the HttpClient built by HttpClientBuilder, even when > everything is turned is not only slower than minimalClient (to be > expected) but also slower than DefaultHttpClient in previous versions. > Not a major difference, but definitely the inverse of what of whats > in the results you sent. > Honestly, there is really no significant differences between the two I can think of. I cannot completely rule out a possibility of some green men from Mars randomly inserting Thread#sleep() statements but I consider it unlikely. I'll see if I can reproduce the issue locally. > I'm still not seeing a major issue with the contention i mentioned i'd > seen in profiler, be an interesting experiment to test performance > with 500 treads, both with/without proxies though. > I'll give it a shot this weekend. Cheers Oleg > Dan > > > oleg@ubuntu:~/src/apache.org/httpcomponents/httpclient-benchmark$ gr -q > > benchmark > > = > > HTTP agent: Apache HttpClient 3.1 > > - > > 100 POST requests > > - > > Document URI: http://localhost:53796/echo > > Document Length:2048 bytes > > > > Concurrency level: 50 > > Time taken for tests: 64.625 seconds > > Complete requests: 100 > > Failed requests:0 > > Content transferred:204800 bytes > > Requests per second:15473.888 [#/sec] (mean) > > - > > = > > HTTP agent: Apache HttpClient (ver: 4.3.3) > > - > > 100 POST requests > > - > > Document URI: http://localhost:47900/echo > > Document Length:2048 bytes > > > > Concurrency level: 50 > > Time taken for tests: 55.837 seconds > > Complete requests: 100 > > Failed requests:0 > > Content transferred:204800 bytes > > Requests per second:17909.271 [#/sec] (mean) > > > > > > > > - > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Wed, 2014-05-14 at 16:05 +0100, Daniel Feist wrote: ... > I agree 10% isn't that much and this testing maybe isn't 100% through, > but I think it's fairly clear that unless you use minimalClient with > 4.3, it doesn't matter which features you turn off (even all of them), > 4.3 still doesn't perform as fast as 4.2. Also I'm surprised that 3.1 > runs so fast, even if it doesn't scale quite as well as 4.x. > > Couple of outstanding question: > > 1) I assume your recommendation about leaving stale connection > checking on by default unless you know what you are doing is still the > same in 4.3? Maybe it's a good idea to turn if off by default for > idempotent http methods and on by default for others, rather than > having it on for everything? > We are actually planning to do away with indiscriminate stale connection checking for each request in 4.4 [1][2] The current plan is to start checking connections only after a certain period of inactivity. We might also treat idempotent requests differently, but generally people tend to scream bloody murder every time they see an I/O exception and blame it on HttpClient. This is the reason why we have the damn check on by default. > 2) I'm still unsure why 4.3 is running slower than 4.2, even with > everything turned off.. > > Let's keep digging. > For my particular use case i think I might just stick to 3.1 for now, > until our next minor release anyway, because the huge performance > difference I was seeing between 3.1 and 4.3 it seems was 100% down to > the stale connection check! That said i like the fluent interface and > flexibility of 4.3. > I would advise against using HC 3.1. It has been unmaintained for too long. HC 4.2 has the same fluent APIs so you should not lose too much. Oleg [1] http://wiki.apache.org/HttpComponents/HttpComponentsRoadmap [2] https://issues.apache.org/jira/browse/HTTPCLIENT-1493 - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
>> For some reason the HttpClient built by HttpClientBuilder, even when >> everything is turned is not only slower than minimalClient (to be >> expected) but also slower than DefaultHttpClient in previous versions. >> Not a major difference, but definitely the inverse of what of whats >> in the results you sent. >> > > I am sorry I cannot reproduce it. I replaced HC in minimal configuration > with this With the configuration for 4.3.3 that you used i get: - Jetty SelectChannelConnector: 32 core: 3.1 is 10% faster. (CPU: 4.3.3-> 1470%, 3.1 -> 1780% ) 4 core: 4.3.3 is 5% faster. - Jetty BlockingChannelConnector 32 core: 3.1 is 12.5% faster (CPU: 4.3.3 -> 1210%, 3.1 -> 1640%) 4 core: 4.3.3 is 30% faster (I ran them separately and not in same gradle run, to be completely fair * I adjusted invocation count to ensure test ran for at least 1 minute) Summary: 1) 3.1 is slightly faster than 4.3.3 when there is no cpu constraint. When using the Jetty BlockingChannelConnector, this can be seen more clearly. 2) When CPU is constrained 4.3.3 runs faster. Using BlockingChannelConnector makes this much more significant. Note: While i didn't test minimalClient again, from earlier testing, this behaviour seems to affect HTTPClientBuilder/InternalHttpClient but not minimalClient. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
> Could you please find out if this difference is consistent regardless of > the CPU core number used by the system? Done, see other email I didn't record all figures, but did run multiple times and records differences. > Honestly, there is really no significant differences between the two I > can think of. I cannot completely rule out a possibility of some green > men from Mars randomly inserting Thread#sleep() statements but I > consider it unlikely. I'll see if I can reproduce the issue locally. I couldn't imagine a difference but in testing on 32-core with BlockingChannelConnector I got: - 3.1 | 75 K TPS - 4.3.3 Minimal / 4.2.6 | 74 K TPS - 4.3.3 DefultHttpClient | 72 K TPS - 4.3.3 Builder (from your test) | 68 K TPS The difference is small, but it's an interesting problem, working out whats going on. 74 vs 75 is irrelevant, but 68 vs. 75 is bigger. >> I'm still not seeing a major issue with the contention i mentioned i'd >> seen in profiler, be an interesting experiment to test performance >> with 500 treads, both with/without proxies though. >> > > I'll give it a shot this weekend. Not sure you'll see any difference is cpu is constrained. If you do do it, I can test on 32-core box for you. Dan - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Sat, 2014-05-17 at 00:13 +0100, Daniel Feist wrote: > > Could you please find out if this difference is consistent regardless of > > the CPU core number used by the system? > > Done, see other email I didn't record all figures, but did run > multiple times and records differences. > > > Honestly, there is really no significant differences between the two I > > can think of. I cannot completely rule out a possibility of some green > > men from Mars randomly inserting Thread#sleep() statements but I > > consider it unlikely. I'll see if I can reproduce the issue locally. > > I couldn't imagine a difference but in testing on 32-core with > BlockingChannelConnector I got: > > - 3.1 | 75 K TPS > - 4.3.3 Minimal / 4.2.6 | 74 K TPS > - 4.3.3 DefultHttpClient | 72 K TPS > - 4.3.3 Builder (from your test) | 68 K TPS > > The difference is small, but it's an interesting problem, working out > whats going on. 74 vs 75 is irrelevant, but 68 vs. 75 is bigger. > > >> I'm still not seeing a major issue with the contention i mentioned i'd > >> seen in profiler, be an interesting experiment to test performance > >> with 500 treads, both with/without proxies though. > >> > > > > I'll give it a shot this weekend. > > Not sure you'll see any difference is cpu is constrained. If you do do > it, I can test on 32-core box for you. > Daniel I removed all dynamic proxies and got approximately 10% better performance with my two CPU core PC. Could you please build HttpClient from my private experimental branch at github and re-run your tests? https://github.com/ok2c/httpclient/tree/dyn_proxies https://github.com/ok2c/httpclient/commit/246095522504ddffe07caf70db4c051fa0d31872 Cheers Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
Sure, which jetty connector? Blocking is better because then test is more about httpclient. With other connector jetty uses more cpu and has more context switching. Dan On 17 May 2014 20:15, "Oleg Kalnichevski" wrote: > On Sat, 2014-05-17 at 00:13 +0100, Daniel Feist wrote: > > > Could you please find out if this difference is consistent regardless > of > > > the CPU core number used by the system? > > > > Done, see other email I didn't record all figures, but did run > > multiple times and records differences. > > > > > Honestly, there is really no significant differences between the two I > > > can think of. I cannot completely rule out a possibility of some green > > > men from Mars randomly inserting Thread#sleep() statements but I > > > consider it unlikely. I'll see if I can reproduce the issue locally. > > > > I couldn't imagine a difference but in testing on 32-core with > > BlockingChannelConnector I got: > > > > - 3.1 | 75 K TPS > > - 4.3.3 Minimal / 4.2.6 | 74 K TPS > > - 4.3.3 DefultHttpClient | 72 K TPS > > - 4.3.3 Builder (from your test) | 68 K TPS > > > > The difference is small, but it's an interesting problem, working out > > whats going on. 74 vs 75 is irrelevant, but 68 vs. 75 is bigger. > > > > >> I'm still not seeing a major issue with the contention i mentioned i'd > > >> seen in profiler, be an interesting experiment to test performance > > >> with 500 treads, both with/without proxies though. > > >> > > > > > > I'll give it a shot this weekend. > > > > Not sure you'll see any difference is cpu is constrained. If you do do > > it, I can test on 32-core box for you. > > > > Daniel > > I removed all dynamic proxies and got approximately 10% better > performance with my two CPU core PC. > > Could you please build HttpClient from my private experimental branch at > github and re-run your tests? > > https://github.com/ok2c/httpclient/tree/dyn_proxies > > https://github.com/ok2c/httpclient/commit/246095522504ddffe07caf70db4c051fa0d31872 > > Cheers > > Oleg > >
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Sat, 2014-05-17 at 22:10 +0100, Daniel Feist wrote: > Sure, which jetty connector? Blocking is better because then test is more > about httpclient. With other connector jetty uses more cpu and has more > context switching. > I used the same micro-benchmark as before, the same jetty connector, the same minimal HC config. I am really sorry but I cannot reproduce any significant difference between minimal HC and the defaul one with performance tuning, and more importantly I see no reason for that difference to be there. Let's see what kind of reasons you are going to get and then proceed from there. Oleg > Dan > On 17 May 2014 20:15, "Oleg Kalnichevski" wrote: > > > On Sat, 2014-05-17 at 00:13 +0100, Daniel Feist wrote: > > > > Could you please find out if this difference is consistent regardless > > of > > > > the CPU core number used by the system? > > > > > > Done, see other email I didn't record all figures, but did run > > > multiple times and records differences. > > > > > > > Honestly, there is really no significant differences between the two I > > > > can think of. I cannot completely rule out a possibility of some green > > > > men from Mars randomly inserting Thread#sleep() statements but I > > > > consider it unlikely. I'll see if I can reproduce the issue locally. > > > > > > I couldn't imagine a difference but in testing on 32-core with > > > BlockingChannelConnector I got: > > > > > > - 3.1 | 75 K TPS > > > - 4.3.3 Minimal / 4.2.6 | 74 K TPS > > > - 4.3.3 DefultHttpClient | 72 K TPS > > > - 4.3.3 Builder (from your test) | 68 K TPS > > > > > > The difference is small, but it's an interesting problem, working out > > > whats going on. 74 vs 75 is irrelevant, but 68 vs. 75 is bigger. > > > > > > >> I'm still not seeing a major issue with the contention i mentioned i'd > > > >> seen in profiler, be an interesting experiment to test performance > > > >> with 500 treads, both with/without proxies though. > > > >> > > > > > > > > I'll give it a shot this weekend. > > > > > > Not sure you'll see any difference is cpu is constrained. If you do do > > > it, I can test on 32-core box for you. > > > > > > > Daniel > > > > I removed all dynamic proxies and got approximately 10% better > > performance with my two CPU core PC. > > > > Could you please build HttpClient from my private experimental branch at > > github and re-run your tests? > > > > https://github.com/ok2c/httpclient/tree/dyn_proxies > > > > https://github.com/ok2c/httpclient/commit/246095522504ddffe07caf70db4c051fa0d31872 > > > > Cheers > > > > Oleg > > > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On May 18, 2014 2:17:00 PM CEST, Oleg Kalnichevski wrote: >On Sat, 2014-05-17 at 22:10 +0100, Daniel Feist wrote: >> Sure, which jetty connector? Blocking is better because then test is >more >> about httpclient. With other connector jetty uses more cpu and has >more >> context switching. >> > >I used the same micro-benchmark as before, the same jetty connector, >the >same minimal HC config. I am really sorry but I cannot reproduce any >significant difference between minimal HC and the defaul one with >performance tuning, and more importantly I see no reason for that >difference to be there. > >Let's see what kind of reasons you are going to get and then proceed >from there. I am such a mess. Reasons -> results. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
Here are the results!! Interesting... I reran everything: - Added JVM options to even out garbage collection. - Ran each test for minutes 5 minutes. - Used Jetty SelectChannelConnector, same as you always used. (other one is faster, but less stable). https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 Configurations of httpClient are as you have in svn accept for 4.3.3 (not minimal) which is configured as follows: The non-minimal configuration for 4.3.3 is: final RequestConfig requestConfig = RequestConfig.custom() .setStaleConnectionCheckEnabled(false) .setExpectContinueEnabled(false) .build(); this.httpclient = HttpClients.custom() .setConnectionManager(this.mgr) .disableContentCompression() .setDefaultRequestConfig(requestConfig) .build(); Dan On Sat, May 17, 2014 at 8:15 PM, Oleg Kalnichevski wrote: > On Sat, 2014-05-17 at 00:13 +0100, Daniel Feist wrote: >> > Could you please find out if this difference is consistent regardless of >> > the CPU core number used by the system? >> >> Done, see other email I didn't record all figures, but did run >> multiple times and records differences. >> >> > Honestly, there is really no significant differences between the two I >> > can think of. I cannot completely rule out a possibility of some green >> > men from Mars randomly inserting Thread#sleep() statements but I >> > consider it unlikely. I'll see if I can reproduce the issue locally. >> >> I couldn't imagine a difference but in testing on 32-core with >> BlockingChannelConnector I got: >> >> - 3.1 | 75 K TPS >> - 4.3.3 Minimal / 4.2.6 | 74 K TPS >> - 4.3.3 DefultHttpClient | 72 K TPS >> - 4.3.3 Builder (from your test) | 68 K TPS >> >> The difference is small, but it's an interesting problem, working out >> whats going on. 74 vs 75 is irrelevant, but 68 vs. 75 is bigger. >> >> >> I'm still not seeing a major issue with the contention i mentioned i'd >> >> seen in profiler, be an interesting experiment to test performance >> >> with 500 treads, both with/without proxies though. >> >> >> > >> > I'll give it a shot this weekend. >> >> Not sure you'll see any difference is cpu is constrained. If you do do >> it, I can test on 32-core box for you. >> > > Daniel > > I removed all dynamic proxies and got approximately 10% better > performance with my two CPU core PC. > > Could you please build HttpClient from my private experimental branch at > github and re-run your tests? > > https://github.com/ok2c/httpclient/tree/dyn_proxies > https://github.com/ok2c/httpclient/commit/246095522504ddffe07caf70db4c051fa0d31872 > > Cheers > > Oleg > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
You'll see 32-core (no cpu saturation) and 4-core (cpu saturated) are in different tabs in spreadsheet.. On Wed, May 21, 2014 at 3:40 PM, Daniel Feist wrote: > Here are the results!! Interesting... > > I reran everything: > - Added JVM options to even out garbage collection. > - Ran each test for minutes 5 minutes. > - Used Jetty SelectChannelConnector, same as you always used. (other > one is faster, but less stable). > > https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 > > Configurations of httpClient are as you have in svn accept for 4.3.3 > (not minimal) which is configured as follows: > > The non-minimal configuration for 4.3.3 is: > final RequestConfig requestConfig = RequestConfig.custom() > .setStaleConnectionCheckEnabled(false) > .setExpectContinueEnabled(false) > .build(); > this.httpclient = HttpClients.custom() > .setConnectionManager(this.mgr) > .disableContentCompression() > .setDefaultRequestConfig(requestConfig) > .build(); > > > Dan > > On Sat, May 17, 2014 at 8:15 PM, Oleg Kalnichevski wrote: >> On Sat, 2014-05-17 at 00:13 +0100, Daniel Feist wrote: >>> > Could you please find out if this difference is consistent regardless of >>> > the CPU core number used by the system? >>> >>> Done, see other email I didn't record all figures, but did run >>> multiple times and records differences. >>> >>> > Honestly, there is really no significant differences between the two I >>> > can think of. I cannot completely rule out a possibility of some green >>> > men from Mars randomly inserting Thread#sleep() statements but I >>> > consider it unlikely. I'll see if I can reproduce the issue locally. >>> >>> I couldn't imagine a difference but in testing on 32-core with >>> BlockingChannelConnector I got: >>> >>> - 3.1 | 75 K TPS >>> - 4.3.3 Minimal / 4.2.6 | 74 K TPS >>> - 4.3.3 DefultHttpClient | 72 K TPS >>> - 4.3.3 Builder (from your test) | 68 K TPS >>> >>> The difference is small, but it's an interesting problem, working out >>> whats going on. 74 vs 75 is irrelevant, but 68 vs. 75 is bigger. >>> >>> >> I'm still not seeing a major issue with the contention i mentioned i'd >>> >> seen in profiler, be an interesting experiment to test performance >>> >> with 500 treads, both with/without proxies though. >>> >> >>> > >>> > I'll give it a shot this weekend. >>> >>> Not sure you'll see any difference is cpu is constrained. If you do do >>> it, I can test on 32-core box for you. >>> >> >> Daniel >> >> I removed all dynamic proxies and got approximately 10% better >> performance with my two CPU core PC. >> >> Could you please build HttpClient from my private experimental branch at >> github and re-run your tests? >> >> https://github.com/ok2c/httpclient/tree/dyn_proxies >> https://github.com/ok2c/httpclient/commit/246095522504ddffe07caf70db4c051fa0d31872 >> >> Cheers >> >> Oleg >> - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: > Here are the results!! Interesting... > > I reran everything: > - Added JVM options to even out garbage collection. > - Ran each test for minutes 5 minutes. > - Used Jetty SelectChannelConnector, same as you always used. (other > one is faster, but less stable). > > https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 > > Configurations of httpClient are as you have in svn accept for 4.3.3 > (not minimal) which is configured as follows: > > The non-minimal configuration for 4.3.3 is: > final RequestConfig requestConfig = RequestConfig.custom() > .setStaleConnectionCheckEnabled(false) > .setExpectContinueEnabled(false) > .build(); > this.httpclient = HttpClients.custom() > .setConnectionManager(this.mgr) > .disableContentCompression() > .setDefaultRequestConfig(requestConfig) > .build(); > > > Dan > Hi Daniel, What I am seeing makes me think we are on the right track, aren't we? I find HC 3.1 performance with 32 CPU cores a little suspect, but HC 3.x has a completely different connection pool manager implementation, so I do not see this as anything worth looking into. I'll merge my changes into the official SVN repository shortly. I personally would consider the issue resolved unless you think there is still something that needs investigation. Cheers Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
You change definitely Improve things :-) unless there are other changes in 4.3.4-SNAPSHOT that may affect performance?? My Observations: - Minimal is 7% faster both before and after your changes. This is fine though, it's doing a bit more. - 4.3 is faster than 3.1 when cpu is constrained (+5%) - 4.3 is slower than 3.1 when cpu is constrained (-5%) - With your changes 4.3 performs same as 3.1 on 32 core and 25% (instead of 5%) faster on 4-core. There are still a few things I don't understand: - I'm unsure why your change to eliminate proxy would have such an impact with the 10 thread case. (+65%), good though. - Unsure why when cpu isn't constrained 3.1 does so well. All I can think of it that 3.1 is more cpu intensive than 4.3 and so doesn't fair so badly when plenty of cpu. Dan On Wed, May 21, 2014 at 4:02 PM, Oleg Kalnichevski wrote: > On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: >> Here are the results!! Interesting... >> >> I reran everything: >> - Added JVM options to even out garbage collection. >> - Ran each test for minutes 5 minutes. >> - Used Jetty SelectChannelConnector, same as you always used. (other >> one is faster, but less stable). >> >> https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 >> >> Configurations of httpClient are as you have in svn accept for 4.3.3 >> (not minimal) which is configured as follows: >> >> The non-minimal configuration for 4.3.3 is: >> final RequestConfig requestConfig = RequestConfig.custom() >> .setStaleConnectionCheckEnabled(false) >> .setExpectContinueEnabled(false) >> .build(); >> this.httpclient = HttpClients.custom() >> .setConnectionManager(this.mgr) >> .disableContentCompression() >> .setDefaultRequestConfig(requestConfig) >> .build(); >> >> >> Dan >> > > Hi Daniel, > > What I am seeing makes me think we are on the right track, aren't we? I > find HC 3.1 performance with 32 CPU cores a little suspect, but HC 3.x > has a completely different connection pool manager implementation, so I > do not see this as anything worth looking into. > > I'll merge my changes into the official SVN repository shortly. I > personally would consider the issue resolved unless you think there is > still something that needs investigation. > > Cheers > > Oleg > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Wed, 2014-05-21 at 16:38 +0100, Daniel Feist wrote: > You change definitely Improve things :-) unless there are other > changes in 4.3.4-SNAPSHOT that may affect performance?? > None I personally know of. > My Observations: > - Minimal is 7% faster both before and after your changes. This is > fine though, it's doing a bit more. > - 4.3 is faster than 3.1 when cpu is constrained (+5%) > - 4.3 is slower than 3.1 when cpu is constrained (-5%) > - With your changes 4.3 performs same as 3.1 on 32 core This seems only to be the case with certain concurrency levels (20 and 200), which is suspicious. > and 25% > (instead of 5%) faster on 4-core. > > There are still a few things I don't understand: > - I'm unsure why your change to eliminate proxy would have such an > impact with the 10 thread case. (+65%), good though. Could be a fluke. I see no good reason for such a significant difference. > - Unsure why when cpu isn't constrained 3.1 does so well. All I can > think of it that 3.1 is more cpu intensive than 4.3 and so doesn't > fair so badly when plenty of cpu. > I think it is a pretty good theory. HC 3.x should be generating more intermediate garbage and taking more CPU cycles while parsing HTTP messages compared to HC 4.x. More CPU cores seem to alleviate this inefficiency. The patch has been committed to SVN trunk and 4.3.x branch. http://svn.apache.org/r1596649 http://svn.apache.org/r1596653 Thank you so much for reporting the problem and helping fix it. Cheers Oleg > Dan > > On Wed, May 21, 2014 at 4:02 PM, Oleg Kalnichevski wrote: > > On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: > >> Here are the results!! Interesting... > >> > >> I reran everything: > >> - Added JVM options to even out garbage collection. > >> - Ran each test for minutes 5 minutes. > >> - Used Jetty SelectChannelConnector, same as you always used. (other > >> one is faster, but less stable). > >> > >> https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 > >> > >> Configurations of httpClient are as you have in svn accept for 4.3.3 > >> (not minimal) which is configured as follows: > >> > >> The non-minimal configuration for 4.3.3 is: > >> final RequestConfig requestConfig = RequestConfig.custom() > >> .setStaleConnectionCheckEnabled(false) > >> .setExpectContinueEnabled(false) > >> .build(); > >> this.httpclient = HttpClients.custom() > >> .setConnectionManager(this.mgr) > >> .disableContentCompression() > >> .setDefaultRequestConfig(requestConfig) > >> .build(); > >> > >> > >> Dan > >> > > > > Hi Daniel, > > > > What I am seeing makes me think we are on the right track, aren't we? I > > find HC 3.1 performance with 32 CPU cores a little suspect, but HC 3.x > > has a completely different connection pool manager implementation, so I > > do not see this as anything worth looking into. > > > > I'll merge my changes into the official SVN repository shortly. I > > personally would consider the issue resolved unless you think there is > > still something that needs investigation. > > > > Cheers > > > > Oleg > > > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
>> - 4.3 is faster than 3.1 when cpu is constrained (+5%) >> - 4.3 is slower than 3.1 when cpu is constrained (-5%) >> - With your changes 4.3 performs same as 3.1 on 32 core > > This seems only to be the case with certain concurrency levels (20 and > 200), which is suspicious. Agree, weird, but as long as 4.x is at least as fast as 3.1 (it is), then there are many other reasons to forget about 3.1. >> - I'm unsure why your change to eliminate proxy would have such an >> impact with the 10 thread case. (+65%), good though. > > Could be a fluke. I see no good reason for such a significant > difference. I doubt it's a fluke was quite clear and repeatable. Not important to understand fully i don't think anyway > The patch has been committed to SVN trunk and 4.3.x branch. > > http://svn.apache.org/r1596649 > http://svn.apache.org/r1596653 > > Thank you so much for reporting the problem and helping fix it. Great! This wasn't a significant problem, just a potential optimisation. Dan > > Cheers > > Oleg > >> Dan >> >> On Wed, May 21, 2014 at 4:02 PM, Oleg Kalnichevski wrote: >> > On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: >> >> Here are the results!! Interesting... >> >> >> >> I reran everything: >> >> - Added JVM options to even out garbage collection. >> >> - Ran each test for minutes 5 minutes. >> >> - Used Jetty SelectChannelConnector, same as you always used. (other >> >> one is faster, but less stable). >> >> >> >> https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 >> >> >> >> Configurations of httpClient are as you have in svn accept for 4.3.3 >> >> (not minimal) which is configured as follows: >> >> >> >> The non-minimal configuration for 4.3.3 is: >> >> final RequestConfig requestConfig = RequestConfig.custom() >> >> .setStaleConnectionCheckEnabled(false) >> >> .setExpectContinueEnabled(false) >> >> .build(); >> >> this.httpclient = HttpClients.custom() >> >> .setConnectionManager(this.mgr) >> >> .disableContentCompression() >> >> .setDefaultRequestConfig(requestConfig) >> >> .build(); >> >> >> >> >> >> Dan >> >> >> > >> > Hi Daniel, >> > >> > What I am seeing makes me think we are on the right track, aren't we? I >> > find HC 3.1 performance with 32 CPU cores a little suspect, but HC 3.x >> > has a completely different connection pool manager implementation, so I >> > do not see this as anything worth looking into. >> > >> > I'll merge my changes into the official SVN repository shortly. I >> > personally would consider the issue resolved unless you think there is >> > still something that needs investigation. >> > >> > Cheers >> > >> > Oleg >> > >> > > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
BTW, is there a jira issue for this I can reference? thanks! On Wed, May 21, 2014 at 8:30 PM, Daniel Feist wrote: >>> - 4.3 is faster than 3.1 when cpu is constrained (+5%) >>> - 4.3 is slower than 3.1 when cpu is constrained (-5%) >>> - With your changes 4.3 performs same as 3.1 on 32 core >> >> This seems only to be the case with certain concurrency levels (20 and >> 200), which is suspicious. > > Agree, weird, but as long as 4.x is at least as fast as 3.1 (it is), > then there are many other reasons to forget about 3.1. > >>> - I'm unsure why your change to eliminate proxy would have such an >>> impact with the 10 thread case. (+65%), good though. >> >> Could be a fluke. I see no good reason for such a significant >> difference. > > I doubt it's a fluke was quite clear and repeatable. Not important to > understand fully i don't think anyway > >> The patch has been committed to SVN trunk and 4.3.x branch. >> >> http://svn.apache.org/r1596649 >> http://svn.apache.org/r1596653 >> >> Thank you so much for reporting the problem and helping fix it. > > Great! This wasn't a significant problem, just a potential optimisation. > > Dan > > >> >> Cheers >> >> Oleg >> >>> Dan >>> >>> On Wed, May 21, 2014 at 4:02 PM, Oleg Kalnichevski wrote: >>> > On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: >>> >> Here are the results!! Interesting... >>> >> >>> >> I reran everything: >>> >> - Added JVM options to even out garbage collection. >>> >> - Ran each test for minutes 5 minutes. >>> >> - Used Jetty SelectChannelConnector, same as you always used. (other >>> >> one is faster, but less stable). >>> >> >>> >> https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 >>> >> >>> >> Configurations of httpClient are as you have in svn accept for 4.3.3 >>> >> (not minimal) which is configured as follows: >>> >> >>> >> The non-minimal configuration for 4.3.3 is: >>> >> final RequestConfig requestConfig = RequestConfig.custom() >>> >> .setStaleConnectionCheckEnabled(false) >>> >> .setExpectContinueEnabled(false) >>> >> .build(); >>> >> this.httpclient = HttpClients.custom() >>> >> .setConnectionManager(this.mgr) >>> >> .disableContentCompression() >>> >> .setDefaultRequestConfig(requestConfig) >>> >> .build(); >>> >> >>> >> >>> >> Dan >>> >> >>> > >>> > Hi Daniel, >>> > >>> > What I am seeing makes me think we are on the right track, aren't we? I >>> > find HC 3.1 performance with 32 CPU cores a little suspect, but HC 3.x >>> > has a completely different connection pool manager implementation, so I >>> > do not see this as anything worth looking into. >>> > >>> > I'll merge my changes into the official SVN repository shortly. I >>> > personally would consider the issue resolved unless you think there is >>> > still something that needs investigation. >>> > >>> > Cheers >>> > >>> > Oleg >>> > >>> > >> >> - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
On Thu, 2014-05-22 at 11:05 +0100, Daniel Feist wrote: > BTW, is there a jira issue for this I can reference? > I did not raise a ticket (I have enough fun raising tickets at my day work). You are welcome to raise one postmortem so to speak, if you feel like. Oleg > thanks! > > On Wed, May 21, 2014 at 8:30 PM, Daniel Feist wrote: > >>> - 4.3 is faster than 3.1 when cpu is constrained (+5%) > >>> - 4.3 is slower than 3.1 when cpu is constrained (-5%) > >>> - With your changes 4.3 performs same as 3.1 on 32 core > >> > >> This seems only to be the case with certain concurrency levels (20 and > >> 200), which is suspicious. > > > > Agree, weird, but as long as 4.x is at least as fast as 3.1 (it is), > > then there are many other reasons to forget about 3.1. > > > >>> - I'm unsure why your change to eliminate proxy would have such an > >>> impact with the 10 thread case. (+65%), good though. > >> > >> Could be a fluke. I see no good reason for such a significant > >> difference. > > > > I doubt it's a fluke was quite clear and repeatable. Not important to > > understand fully i don't think anyway > > > >> The patch has been committed to SVN trunk and 4.3.x branch. > >> > >> http://svn.apache.org/r1596649 > >> http://svn.apache.org/r1596653 > >> > >> Thank you so much for reporting the problem and helping fix it. > > > > Great! This wasn't a significant problem, just a potential optimisation. > > > > Dan > > > > > >> > >> Cheers > >> > >> Oleg > >> > >>> Dan > >>> > >>> On Wed, May 21, 2014 at 4:02 PM, Oleg Kalnichevski > >>> wrote: > >>> > On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: > >>> >> Here are the results!! Interesting... > >>> >> > >>> >> I reran everything: > >>> >> - Added JVM options to even out garbage collection. > >>> >> - Ran each test for minutes 5 minutes. > >>> >> - Used Jetty SelectChannelConnector, same as you always used. (other > >>> >> one is faster, but less stable). > >>> >> > >>> >> https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 > >>> >> > >>> >> Configurations of httpClient are as you have in svn accept for 4.3.3 > >>> >> (not minimal) which is configured as follows: > >>> >> > >>> >> The non-minimal configuration for 4.3.3 is: > >>> >> final RequestConfig requestConfig = RequestConfig.custom() > >>> >> .setStaleConnectionCheckEnabled(false) > >>> >> .setExpectContinueEnabled(false) > >>> >> .build(); > >>> >> this.httpclient = HttpClients.custom() > >>> >> .setConnectionManager(this.mgr) > >>> >> .disableContentCompression() > >>> >> .setDefaultRequestConfig(requestConfig) > >>> >> .build(); > >>> >> > >>> >> > >>> >> Dan > >>> >> > >>> > > >>> > Hi Daniel, > >>> > > >>> > What I am seeing makes me think we are on the right track, aren't we? I > >>> > find HC 3.1 performance with 32 CPU cores a little suspect, but HC 3.x > >>> > has a completely different connection pool manager implementation, so I > >>> > do not see this as anything worth looking into. > >>> > > >>> > I'll merge my changes into the official SVN repository shortly. I > >>> > personally would consider the issue resolved unless you think there is > >>> > still something that needs investigation. > >>> > > >>> > Cheers > >>> > > >>> > Oleg > >>> > > >>> > > >> > >> > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Thread Contention (due to use of proxys) in HttpClient 4.3
No worries. thanks! On Thu, May 22, 2014 at 1:17 PM, Oleg Kalnichevski wrote: > On Thu, 2014-05-22 at 11:05 +0100, Daniel Feist wrote: >> BTW, is there a jira issue for this I can reference? >> > > I did not raise a ticket (I have enough fun raising tickets at my day > work). You are welcome to raise one postmortem so to speak, if you feel > like. > > Oleg > >> thanks! >> >> On Wed, May 21, 2014 at 8:30 PM, Daniel Feist wrote: >> >>> - 4.3 is faster than 3.1 when cpu is constrained (+5%) >> >>> - 4.3 is slower than 3.1 when cpu is constrained (-5%) >> >>> - With your changes 4.3 performs same as 3.1 on 32 core >> >> >> >> This seems only to be the case with certain concurrency levels (20 and >> >> 200), which is suspicious. >> > >> > Agree, weird, but as long as 4.x is at least as fast as 3.1 (it is), >> > then there are many other reasons to forget about 3.1. >> > >> >>> - I'm unsure why your change to eliminate proxy would have such an >> >>> impact with the 10 thread case. (+65%), good though. >> >> >> >> Could be a fluke. I see no good reason for such a significant >> >> difference. >> > >> > I doubt it's a fluke was quite clear and repeatable. Not important to >> > understand fully i don't think anyway >> > >> >> The patch has been committed to SVN trunk and 4.3.x branch. >> >> >> >> http://svn.apache.org/r1596649 >> >> http://svn.apache.org/r1596653 >> >> >> >> Thank you so much for reporting the problem and helping fix it. >> > >> > Great! This wasn't a significant problem, just a potential optimisation. >> > >> > Dan >> > >> > >> >> >> >> Cheers >> >> >> >> Oleg >> >> >> >>> Dan >> >>> >> >>> On Wed, May 21, 2014 at 4:02 PM, Oleg Kalnichevski >> >>> wrote: >> >>> > On Wed, 2014-05-21 at 15:40 +0100, Daniel Feist wrote: >> >>> >> Here are the results!! Interesting... >> >>> >> >> >>> >> I reran everything: >> >>> >> - Added JVM options to even out garbage collection. >> >>> >> - Ran each test for minutes 5 minutes. >> >>> >> - Used Jetty SelectChannelConnector, same as you always used. (other >> >>> >> one is faster, but less stable). >> >>> >> >> >>> >> https://docs.google.com/a/mulesoft.com/spreadsheets/d/1j2TCeAQmrkWLZyQfetzmtqmUDaGQBiDAr20hC9gEfYE/edit#gid=590558615 >> >>> >> >> >>> >> Configurations of httpClient are as you have in svn accept for 4.3.3 >> >>> >> (not minimal) which is configured as follows: >> >>> >> >> >>> >> The non-minimal configuration for 4.3.3 is: >> >>> >> final RequestConfig requestConfig = RequestConfig.custom() >> >>> >> .setStaleConnectionCheckEnabled(false) >> >>> >> .setExpectContinueEnabled(false) >> >>> >> .build(); >> >>> >> this.httpclient = HttpClients.custom() >> >>> >> .setConnectionManager(this.mgr) >> >>> >> .disableContentCompression() >> >>> >> .setDefaultRequestConfig(requestConfig) >> >>> >> .build(); >> >>> >> >> >>> >> >> >>> >> Dan >> >>> >> >> >>> > >> >>> > Hi Daniel, >> >>> > >> >>> > What I am seeing makes me think we are on the right track, aren't we? I >> >>> > find HC 3.1 performance with 32 CPU cores a little suspect, but HC 3.x >> >>> > has a completely different connection pool manager implementation, so I >> >>> > do not see this as anything worth looking into. >> >>> > >> >>> > I'll merge my changes into the official SVN repository shortly. I >> >>> > personally would consider the issue resolved unless you think there is >> >>> > still something that needs investigation. >> >>> > >> >>> > Cheers >> >>> > >> >>> > Oleg >> >>> > >> >>> > >> >> >> >> >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
