Hi, I spent a good part of yesterday testing and comparing HttpClient performance and the results are interesting: HttpClient 3.1 is 20% faster than HttpClient 4.3 when both configured in the same way and when using 50 client threads!
https://docs.google.com/a/mulesoft.com/spreadsheets/d/1Dqp2dH7K8nQeBzAC3S4hboYmSm67GXWO6AYgTYiVInQ/edit#gid=0 When profiling each version when running with 200 threads, this is what I see: HttpClient4.3: - Significant amount of contention due to creation of proxies as mentioned before. HttpClient4.2 -> - No blocked threads HttpClient3.1: -> - Very significant amount of contention due to synchronized method: org.apache.commons.httpclient.params.DefaultHttpParams.getParameter(String) used multiple times for each request. - Minimal contention getting/returning connections from pool. I think it's clear to see why 4.2 scale a bit better, but I'm unsure why 4.3 (when not using minimal) would perform worse than 4.3.. are there additional features in 4.3? Notes: --------- - I used your http client benchmark project referenced by Oleg - I ran tests on Amazon XL (32-core) VM, ensuring each test ran for 30s+. - I tweaked the Jetty configuration to make jetty perform slightly better (see below). I verified changes were faster by testing before and after this change. - For each version of HttpClient i configured the same bufferSize, soTimeout and disabled stale connection checking. - Also recorded as the values for HttpClient4.3 minimal client. Be interested to hear you thoughts, and let me know if you'd like any more information. thanks, Dan JettyConfig: final BlockingChannelConnector connector = new BlockingChannelConnector(); final Server server = new Server(); server.addConnector(connector); server.setHandler(new RandomDataHandler()); server.setThreadPool(new org.eclipse.jetty.util.thread.QueuedThreadPool(500)); > Jaikit & Daniel, > > Dynamic proxies help eliminate a great deal of unnecessary code but I > have no problem with replacing them with hand crafted classes in order > to reduce lock contention. > > Feel free to raise an enhancement request and contribute patches. > > Oleg > >> On Monday, May 12, 2014 11:17 AM, Oleg Kalnichevski <ol...@apache.org> wrote: >> >> On Mon, 2014-05-12 at 13:21 +0100, Daniel Feist wrote: >> > >> 2) Can I expect 4.2 to scale better? >> > > >> > > I do not think so. In my tests HC 4.3 performs better than 4.2. There >> > > have also been reports >> > >> > Even with high concurrency of say 200 and high TPS, for example in a >> > http proxy scenario? >> > >> >> Yes, also with over 200 concurrent connections. However, for HTTP proxy >> scenarios one may be advised to consider using a non-blocking model. >> >> > >> 3) Is there a way of configuring 4.3.3 to not use proxys? >> > > >> > > No, there is not. The cost of dynamic proxies in modern JREs is believed >> > > to be low. >> > >> > The problem I see isn't the use of proxies, it is in the creation of >> > these proxy instances which is synchnorized on a HashMap 'cache' in >> > two places in java.lang.reflect.Proxy.getProxyClass0(ClassLoader, >> > Class<?>...). This is visible in the profiler screenshot I shared >> > too. >> > >> > This probably isn't an issue with low concurrency, or high concurrency >> > and low TPS, but in my testing/profiling it comes up as an issue. >> > >> >> I use this micro-benchmark to benchmark HttpClient performance between >> versions as well compared to other implementations. >> >> http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore >> >> I am open to the idea of replacing dynamic proxies with hand crafted >> classes, but I would like to see more evidence that >> java.lang.reflect.Proxy is indeed a bottleneck. >> >> Oleg >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
