Thanks for everyonce's concern and input for this question. Now we have confirmed that the problem is in the load balancer side. The pooled http client simply magnified their problem! Thansk a lot!
bit1...@163.com From: bit1...@163.com Date: 2015-07-09 18:02 To: bit1...@163.com; httpclient-users Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3 Sorry for the noise. I found it, it is in a separate maven module. <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>fluent-hc</artifactId> <version>4.5</version> </dependency> bit1...@163.com From: bit1...@163.com Date: 2015-07-09 17:54 To: httpclient-users Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3 Hi, Stefan, You mentioned https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html below. Some code snippet is as follows from the guide. I am using 4.3.6, I would ask under which package of Request class, I didn't find it, Thanks! Request.Get("http://somehost/";) .connectTimeout(1000) .socketTimeout(1000) .execute().returnContent().asString(); bit1...@163.com From: Stefan Magnus Landrø Date: 2015-07-08 15:29 To: HttpClient User Discussion Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3 Few things here: HttpClientPoolUtilsTest: why not use a logging framework instead (log4j, logback, commons logging etc) of creating log files manually? Why all this static code? HttpClientManager Line 122: this willl invoke getHttpclient that will build a brand new client on every time. Most inefficient. You should instead use a singleton pattern: public class HttpUtil { private CloseableHttpClient client = // ... build it in constructor public String execute(String url) { client.execute( ... ) } } Maybe https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html is a better fit for you? 2015-07-08 8:49 GMT+02:00 bit1...@163.com <bit1...@163.com>: > Thanks Alexey for your suggestion. > I have created a maven project and it is here : > https://github.com/bit1129/bit-repo/tree/master/HttpClientUsage > > Thanks! > > > > bit1...@163.com > > From: Alexey Panchenko > Date: 2015-07-08 14:05 > To: HttpClient User Discussion > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of > HttpClient3 > I would suggest going further and creating 2 buildable projects with maven > poms. So one can easily run it. > On Jul 8, 2015 11:57 AM, "bit1...@163.com" <bit1...@163.com> wrote: > > > Ok, thanks Stefan! > > > > The github repo is https://github.com/bit1129/bit-repo . That are 5 java > > files there with the discription (Http Client 3.x and Http Client 4.3.6 > > problematic code) > > > > Thanks very much for you guys's time on this!! > > > > > > > > > > > > bit1...@163.com > > > > From: Stefan Magnus Landrø > > Date: 2015-07-08 12:58 > > To: HttpClient User Discussion > > Subject: Re: httpclient4 is extremely slow than correpsoning code of > > HttpClient3 > > Can you create a github repo with your code? Reviewing gets easier then > > > > Sendt fra min iPhone > > > > > Den 8. jul. 2015 kl. 06.03 skrev "bit1...@163.com" <bit1...@163.com>: > > > > > > Thanks Alexey for the apply。 I tried it, and sorry that it still > doesn't > > work. I am using one thread to sending http request for 1000 times every > 20 > > milliseconds. Some response time are as follows: > > > As you can see, the response time is not stable at all, (there are > > ocurrences with 7~10 seconds). If I don't use pool connection manager, > then > > the response time are very stable, about 200 milliseconds for each > request. > > > > > > Is there code example that issuing HTTP request with pool connection > > manager and apply the best practices for the CloseableHttpClient/response > > release/close or related things. So that, I can compare it with mine, and > > hopefully find whether the problem is. > > > Thanks. > > > > > > My testing code snippet: > > > > > > public void test() { > > > long allStart = System.currentTimeMillis(); > > > while (i++ < loop) { > > > long start = System.currentTimeMillis(); > > > HttpUtils.httpInvoke(TEST_URL, null, null); //This the core code that > > issuing http request. > > > long a = System.currentTimeMillis() - start; > > > String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a); > > > writeToFile(timeSpent + "\n", logFile); > > > Thread.sleep(loopInterval); > > > } > > > writeToFile("Total time spent: " + (System.currentTimeMillis() - > > allStart) + "\n", logFile); > > > > > > > > > } > > > > > > 26 > > > 30 > > > 235 > > > 439 > > > 851 > > > 1667 > > > 1040 > > > 234 > > > 439 > > > 851 > > > 1671 > > > 3310 > > > 25 > > > 27 > > > 26 > > > 238 > > > 25 > > > 234 > > > 435 > > > 24 > > > 235 > > > 24 > > > 26 > > > 27 > > > 25 > > > 232 > > > 27 > > > 26 > > > 37 > > > 29 > > > 28 > > > 251 > > > 472 > > > 915 > > > 1804 > > > 3580 > > > 7132 > > > 10057 > > > 28 > > > 26 > > > 253 > > > 26 > > > 25 > > > 23 > > > 243 > > > 24 > > > 24 > > > 27 > > > 26 > > > 1025 > > > 27 > > > 29 > > > 24 > > > 23 > > > > > > > > > > > > > > > bit1...@163.com > > > > > > From: Alexey Panchenko > > > Date: 2015-07-08 11:42 > > > To: HttpClient User Discussion > > > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code > of > > HttpClient3 > > > AFAIK there is no need in creating a new HttpClient instance every > time, > > it > > > should be created once and reused. > > > > > >> On Wed, Jul 8, 2015 at 9:29 AM, bit1...@163.com <bit1...@163.com> > > wrote: > > >> > > >> One more thing that I noticed is: > > >> > > >> When I constructing the CloseableHttpClient, response time gets back > to > > >> normal if I don't pass the PoolingHttpClientConnectionManager object > to > > the > > >> HttpClientBuilder > > >> > > >> HttpClientBuilder httpClientBuilder = HttpClients.custom(); > > >> > > >> /*comment this code so that no PoolingHttpClientConnectionManager > will > > >> be used!*/ > > >> //httpClientBuilder.setConnectionManager(cm); > > >> > > >> //other configurations > > >> return httpClientBuilder.build(); > > >> > > >> Seems that there are something related with > > >> PoolingHttpClientConnectionManager that cause the request's response > > time > > >> is abnormally long. > > >> > > >> Here is my code about PoolingHttpClientConnectionManager : > > >> > > >> cm = new PoolingHttpClientConnectionManager(); > > >> cm.setMaxTotal(1024); > > >> cm.setDefaultMaxPerRoute(128); > > >> > > >> > > >> > > >> bit1...@163.com > > >> > > >> From: bit1...@163.com > > >> Date: 2015-07-08 10:25 > > >> To: httpclient-users > > >> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code > of > > >> HttpClient3 > > >> Thanks Stefan for the reply. > > >> > > >> Do you mean that I move the CloseableHttpClient out of the try block > > like > > >> following, > > >> > > >> public String execute(HttpGet httpGet) { > > >> String body = ""; > > >> CloseableHttpClient httpclient = this.getHttpclient(); > > >> try{ > > >> CloseableHttpResponse response = httpclient.execute(httpGet); > > >> int status = response.getStatusLine().getStatusCode(); > > >> ///other codes goes for consuming response > > >> > > >> Unfortunately, It still doesn't work for me, and I still see that the > > >> request time is abnormally long. > > >> > > >> One thing I notice, is that when I set the socket time out to be 10 > > >> seconds, there are couple of requests throw exception due to socket > time > > >> out. So I wonder what may cause socket time out issue. With > > httpclient3, I > > >> see no such problems, and the longest request time is no more that 2 > > >> seconds. > > >> > > >> > > >> > > >> > > >> > > >> > > >> bit1...@163.com > > >> From: Stefan Magnus Landrø > > >> Date: 2015-07-08 00:33 > > >> To: HttpClient User Discussion > > >> Subject: Re: httpclient4 is extremely slow than correpsoning code of > > >> HttpClient3 > > >>>> try{ > > >>>> CloseableHttpClient httpclient = > > >> Try with resources will close client > > >> Sendt fra min iPhone > > >>> Den 7. jul. 2015 kl. 16.59 skrev Todd <bit1...@163.com>: > > >>> > > >>> Thanks Stefan for the reply. Could you please point to me which code > > you > > >> are referring to? > > >>> > > >>> > > >>> > > >>> At 2015-07-07 22:49:28, "Stefan Magnus Landrø" < > > stefan.lan...@gmail.com> > > >> wrote: > > >>>> You shouldn't close http client on every request, only response > > >>>> > > >>>> Sendt fra min iPhone > > >>>> > > >>>>> Den 7. jul. 2015 kl. 16.05 skrev "bit1...@163.com" < > bit1...@163.com > > >: > > >>>>> > > >>>>> > > >>>>> Can someone kindly help me on this? Thanks a lot! > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> bit1...@163.com > > >>>>> > > >>>>> From: bit1...@163.com > > >>>>> Date: 2015-07-07 20:51 > > >>>>> To: httpclient-users > > >>>>> Subject: httpclient4 is extremely slow than correpsoning code of > > >> HttpClient3 > > >>>>> > > >>>>> I have following http client code(HttpUtils and > > HttpClientManager)with > > >> HttpClient 4.3.6, and a httpclient code in 3.1, and a test case as > > well. I > > >> observed that HttpClient 4.3.6 is very much slowly than HttpClient 3.1 > > >> code. In HttpClient 4.3.6 version code, there are a lot > > >>>>> socket time out error(the socket time is 10 seconds). > > >>>>> > > >>>>> I pasted the related classes and code below. It is kind of long > code, > > >> but I still would ask you do me a favor to review the http client > > >> configuration that may cause the problem. Many Thanks in advance. > > >>>>> > > >>>>> Following codes include: > > >>>>> 1. HttpClient 4.3.6 code to issue HttpGet request > > >>>>> 2. HttpClient 3.1 code to issue HttpGet request > > >>>>> 3. Test Case that demontrate the problem. > > >>>>> > > >>>>> > > >>>>> ############################################Http Client 4.3.6 Code > > >> goes here############################################ > > >> > > > ///////////////////////////HttpUtils////////////////////////////////////////// > > >>>>> import java.util.ArrayList; > > >>>>> import java.util.List; > > >>>>> import java.util.Map; > > >>>>> > > >>>>> import org.apache.http.Consts; > > >>>>> import org.apache.http.NameValuePair; > > >>>>> import org.apache.http.client.config.RequestConfig; > > >>>>> import org.apache.http.client.entity.UrlEncodedFormEntity; > > >>>>> import org.apache.http.client.methods.HttpGet; > > >>>>> import org.apache.http.client.methods.HttpPost; > > >>>>> import org.apache.http.message.BasicNameValuePair; > > >>>>> import org.slf4j.Logger; > > >>>>> import org.slf4j.LoggerFactory; > > >>>>> > > >>>>> > > >>>>> > > >>>>> public class HttpUtils { > > >>>>> > > >>>>> ///This is the util method that will be used to issue http request. > > >>>>> public static String httpInvoke(String httpUrl, Map<String, Object> > > >> parameters, RequestConfig config){ > > >>>>> HttpClientManager httpClientConnectionManager = > > >> HttpClientManager.getHttpClientConnectionManagerInstance(); > > >>>>> HttpGet httpGet = new HttpGet(httpUrl); > > >>>>> if(config != null ){ > > >>>>> httpGet.setConfig(config); > > >>>>> } > > >>>>> String result = httpClientConnectionManager.execute(httpGet); > > >>>>> return result; > > >>>>> } > > >>>>> } > > >> > > > //////////////////////////////////HttpClientManager///////////////////////////////// > > >>>>> > > >>>>> import java.nio.charset.CodingErrorAction; > > >>>>> import java.util.ArrayList; > > >>>>> import java.util.Collection; > > >>>>> > > >>>>> import org.apache.commons.httpclient.Header; > > >>>>> import org.apache.commons.httpclient.HttpStatus; > > >>>>> import org.apache.commons.lang3.StringUtils; > > >>>>> import org.apache.http.Consts; > > >>>>> import org.apache.http.HttpEntity; > > >>>>> import org.apache.http.HttpHost; > > >>>>> import org.apache.http.client.config.RequestConfig; > > >>>>> import org.apache.http.client.entity.GzipDecompressingEntity; > > >>>>> import org.apache.http.client.methods.CloseableHttpResponse; > > >>>>> import org.apache.http.client.methods.HttpGet; > > >>>>> import org.apache.http.client.methods.HttpPost; > > >>>>> import org.apache.http.config.ConnectionConfig; > > >>>>> import org.apache.http.config.MessageConstraints; > > >>>>> import org.apache.http.config.SocketConfig; > > >>>>> import org.apache.http.impl.client.CloseableHttpClient; > > >>>>> import org.apache.http.impl.client.HttpClientBuilder; > > >>>>> import org.apache.http.impl.client.HttpClients; > > >>>>> import > org.apache.http.impl.conn.PoolingHttpClientConnectionManager; > > >>>>> import org.apache.http.message.BasicHeader; > > >>>>> import org.apache.http.params.CoreProtocolPNames; > > >>>>> import org.apache.http.util.EntityUtils; > > >>>>> import org.slf4j.Logger; > > >>>>> import org.slf4j.LoggerFactory; > > >>>>> > > >>>>> > > >>>>> ///Main class that encapsulate the Http Client > > >>>>> public class HttpClientManager { > > >>>>> private static Logger logger = > > >> LoggerFactory.getLogger(HttpClientManager.class); > > >>>>> > > >>>>> private static int defaultConnectionTimeout = 10*1000; > //connection > > >> timeout > > >>>>> private static int defaultSocketTimeout = 10*1000; //socket time > out > > >>>>> private static int connectionRequestTimeout = 10*1000; > //connection > > >> request timeout > > >>>>> > > >>>>> private static int defaultMaxRouteConnections = 128; > > >>>>> private static int defaultMaxTotalConnections = 1024; > > >>>>> > > >>>>> private static int defaultMaxHeaderCount = 200; > > >>>>> private static int defaultMaxLineLength = 2000; > > >>>>> > > >>>>> private static String Charset = "utf-8"; > > >>>>> > > >>>>> private String proxyHost = null; > > >>>>> private String proxyPort =null; > > >>>>> > > >>>>> private PoolingHttpClientConnectionManager cm; > > >>>>> > > >>>>> private final static HttpClientManager httpClientConnectionManager > = > > >> new HttpClientManager(); > > >>>>> > > >>>>> private HttpClientManager() { > > >>>>> logger.info("HttpClientManager initial!"); > > >>>>> cm = new PoolingHttpClientConnectionManager(); > > >>>>> cm.setMaxTotal(defaultMaxTotalConnections); > > >>>>> cm.setDefaultMaxPerRoute(defaultMaxRouteConnections); > > >>>>> } > > >>>>> > > >>>>> > > >>>>> private CloseableHttpClient getHttpclient(){ > > >>>>> HttpClientBuilder httpClientBuilder = HttpClients.custom(); > > >>>>> httpClientBuilder.setConnectionManager(cm); > > >>>>> > > >>>>> RequestConfig.Builder requestConfigBuilder = RequestConfig.custom() > > >>>>> .setConnectTimeout(defaultConnectionTimeout) > > >>>>> .setSocketTimeout(defaultSocketTimeout) > > >>>>> .setConnectionRequestTimeout(connectionRequestTimeout) > > >>>>> .setExpectContinueEnabled(false) > > >>>>> .setStaleConnectionCheckEnabled(true); > > >>>>> > > >>>>> if(StringUtils.isNotBlank(proxyHost) && > > >> StringUtils.isNotBlank(proxyPort)){ > > >>>>> try{ > > >>>>> logger.info("using proxy, proxyHost:{}, proxyPort:{}", proxyHost, > > >> proxyPort); > > >>>>> int proxyPortInt = Integer.parseInt(proxyPort); > > >>>>> requestConfigBuilder.setProxy(new HttpHost(proxyHost, > proxyPortInt)); > > >>>>> } catch(Exception e){ > > >>>>> logger.error("parseInt proxyPort:{}", proxyPort, e); > > >>>>> } > > >>>>> } > > >>>>> > > >>>>> SocketConfig socketConfig = > > >> SocketConfig.custom().setTcpNoDelay(true).build(); > > >>>>> > > >>>>> MessageConstraints messageConstraints = > > >> > > > MessageConstraints.custom().setMaxHeaderCount(defaultMaxHeaderCount).setMaxLineLength(defaultMaxLineLength).build(); > > >>>>> > > >>>>> ConnectionConfig connectionConfig = ConnectionConfig.custom() > > >>>>> .setMalformedInputAction(CodingErrorAction.IGNORE) > > >>>>> .setUnmappableInputAction(CodingErrorAction.IGNORE) > > >>>>> .setCharset(Consts.UTF_8) > > >>>>> .setMessageConstraints(messageConstraints).build(); > > >>>>> > > >>>>> Collection<BasicHeader> collection = new ArrayList<BasicHeader>(); > > >>>>> collection.add(new BasicHeader("User-Agent", "Mozilla/5.0 (Windows; > > U; > > >> Windows NT 5.1; zh-CN; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3")); > > >>>>> collection.add(new BasicHeader("Accept-Language", > > >> "zh-cn,zh,en-US,en;q=0.5")); > > >>>>> collection.add(new BasicHeader("Accept-Charset", Charset)); > > >>>>> collection.add(new BasicHeader("Accept-Encoding", "gzip")); > > >> > httpClientBuilder.setDefaultRequestConfig(requestConfigBuilder.build()); > > >>>>> httpClientBuilder.setDefaultSocketConfig(socketConfig); > > >>>>> httpClientBuilder.setDefaultConnectionConfig(connectionConfig); > > >>>>> httpClientBuilder.setDefaultHeaders(collection); > > >>>>> > > >>>>> return httpClientBuilder.build(); > > >>>>> } > > >>>>> > > >>>>> ///This is the method that will be call the execute the HttpGet > > request > > >>>>> public String execute(HttpGet httpGet) { > > >>>>> String body = ""; > > >>>>> try{ > > >>>>> CloseableHttpClient httpclient = this.getHttpclient(); > > >>>>> CloseableHttpResponse response = httpclient.execute(httpGet); > > >>>>> int status = response.getStatusLine().getStatusCode(); > > >>>>> try { > > >>>>> if (status == HttpStatus.SC_OK) { > > >>>>> HttpEntity entity = response.getEntity(); > > >>>>> if (entity != null) { > > >>>>> Header header = (Header) entity.getContentEncoding(); > > >>>>> if(header != null && "gzip".equals(header.getValue())){ > > >>>>> body = EntityUtils.toString(new GzipDecompressingEntity(entity), > > >> Charset); > > >>>>> } else { > > >>>>> body = EntityUtils.toString(entity, Charset); > > >>>>> } > > >>>>> } > > >>>>> } else { > > >>>>> logger.error("[httpClientManager] [fail] [httpGet:{}] [status:{}]", > > >> httpGet, status); > > >>>>> } > > >>>>> } finally { > > >>>>> response.close(); > > >>>>> } > > >>>>> } catch(Exception e) { > > >>>>> logger.error("[module:httpClientManager] [action:execute] > > [httpGet:{}] > > >> [error:{}] ", httpGet, e.getMessage(), e); > > >>>>> } > > >>>>> return body; > > >>>>> } > > >>>>> > > >>>>> > > >>>>> ////Singleton object that will be used to access > > >> httpClientConnectionManager > > >>>>> public static HttpClientManager > > >> getHttpClientConnectionManagerInstance(){ > > >>>>> return httpClientConnectionManager; > > >>>>> } > > >>>>> > > >>>>> } > > >>>>> > > >>>>> > > >>>>> ##################################################My HttpClient 1.3 > > >> code goes here########################################### > > >>>>> > > >>>>> import org.apache.commons.httpclient.HttpClient; > > >>>>> import org.apache.commons.httpclient.HttpMethod; > > >>>>> import org.apache.commons.httpclient.methods.GetMethod; > > >>>>> > > >>>>> import java.io.BufferedReader; > > >>>>> import java.io.IOException; > > >>>>> import java.io.InputStream; > > >>>>> import java.io.InputStreamReader; > > >>>>> > > >>>>> public class HttpClientUtil { > > >>>>> > > >>>>> private static final HttpClientUtil INSTANCE = new > HttpClientUtil(); > > >>>>> > > >>>>> private static final int TIMEOUT = 10 * 1000; //10s > > >>>>> > > >>>>> private HttpClientUtil() { > > >>>>> } > > >>>>> > > >>>>> ///Actually, this class is more like a static class than Singleton > > >>>>> public static HttpClientUtil getInstance() { > > >>>>> return INSTANCE; > > >>>>> } > > >>>>> > > >>>>> //brand new http client per request > > >>>>> private HttpClient newHttpClient() { > > >>>>> HttpClient client = new HttpClient(); > > >> > > > client.getHttpConnectionManager().getParams().setConnectionTimeout(TIMEOUT); > > >>>>> > client.getHttpConnectionManager().getParams().setSoTimeout(TIMEOUT); > > >>>>> return client; > > >>>>> } > > >>>>> > > >>>>> //FIXME The encoding should be provided when convert the response > > >> binary stream into string > > >>>>> public static String responseBodyAsString(HttpMethod method) throws > > >> IOException { > > >>>>> BufferedReader br = null; > > >>>>> String lsr = System.getProperty("line.separator"); > > >>>>> try { > > >>>>> InputStream in = method.getResponseBodyAsStream(); > > >>>>> br = new BufferedReader(new InputStreamReader(in, "UTF-8")); > > >>>>> StringBuffer sb = new StringBuffer(); > > >>>>> String line; > > >>>>> while ((line = br.readLine()) != null) { > > >>>>> sb.append(line).append(lsr); > > >>>>> } > > >>>>> return sb.toString(); > > >>>>> } finally { > > >>>>> if (br != null) { > > >>>>> br.close(); > > >>>>> } > > >>>>> } > > >>>>> } > > >>>>> > > >>>>> /////This is the method that will HttpGet to the url > > >>>>> public String get(String url) throws IOException { > > >>>>> GetMethod pm = new GetMethod(url); > > >>>>> pm.setRequestHeader("Connection", "close"); > > >>>>> HttpClient client = newHttpClient(); > > >>>>> try { > > >>>>> client.executeMethod(pm); > > >>>>> String response = responseBodyAsString(pm); > > >>>>> return response; > > >>>>> } finally { > > >>>>> pm.releaseConnection(); > > >>>>> } > > >>>>> } > > >>>>> } > > >>>>> > > >>>>> ##########################################Test > > >> Case########################################## > > >>>>> The following code runs in the one thread in the junit test > > >>>>> while (i++ < 5000) { > > >>>>> long start = System.currentTimeMillis(); > > >>>>> HttpUtils.httpInvoke(TEST_URL, null, null); > > >>>>> long a = System.currentTimeMillis() - start; > > >>>>> String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a); > > >>>>> writeToFile(timeSpent + "\n", logFile); > > >>>>> Thread.sleep(loopInterval); > > >>>>> } > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> bit1...@163.com > > >>>> > > >>>> > --------------------------------------------------------------------- > > >>>> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > >>>> For additional commands, e-mail: > httpclient-users-h...@hc.apache.org > > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > > -- BEKK Open http://open.bekk.no TesTcl - a unit test framework for iRules http://testcl.com
