Thanks Alexey for the apply。 I tried it, and sorry that it still doesn't work.
I am using one thread to sending http request for 1000 times every 20
milliseconds. Some response time are as follows:
As you can see, the response time is not stable at all, (there are ocurrences
with 7~10 seconds). If I don't use pool connection manager, then the response
time are very stable, about 200 milliseconds for each request.
Is there code example that issuing HTTP request with pool connection manager
and apply the best practices for the CloseableHttpClient/response release/close
or related things. So that, I can compare it with mine, and hopefully find
whether the problem is.
Thanks.
My testing code snippet:
public void test() {
long allStart = System.currentTimeMillis();
while (i++ < loop) {
long start = System.currentTimeMillis();
HttpUtils.httpInvoke(TEST_URL, null, null); //This the core code that issuing
http request.
long a = System.currentTimeMillis() - start;
String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
writeToFile(timeSpent + "\n", logFile);
Thread.sleep(loopInterval);
}
writeToFile("Total time spent: " + (System.currentTimeMillis() - allStart) +
"\n", logFile);
}
26
30
235
439
851
1667
1040
234
439
851
1671
3310
25
27
26
238
25
234
435
24
235
24
26
27
25
232
27
26
37
29
28
251
472
915
1804
3580
7132
10057
28
26
253
26
25
23
243
24
24
27
26
1025
27
29
24
23
bit1...@163.com
From: Alexey Panchenko
Date: 2015-07-08 11:42
To: HttpClient User Discussion
Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
HttpClient3
AFAIK there is no need in creating a new HttpClient instance every time, it
should be created once and reused.
On Wed, Jul 8, 2015 at 9:29 AM, bit1...@163.com <bit1...@163.com> wrote:
> One more thing that I noticed is:
>
> When I constructing the CloseableHttpClient, response time gets back to
> normal if I don't pass the PoolingHttpClientConnectionManager object to the
> HttpClientBuilder
>
> HttpClientBuilder httpClientBuilder = HttpClients.custom();
>
> /*comment this code so that no PoolingHttpClientConnectionManager will
> be used!*/
> //httpClientBuilder.setConnectionManager(cm);
>
> //other configurations
> return httpClientBuilder.build();
>
> Seems that there are something related with
> PoolingHttpClientConnectionManager that cause the request's response time
> is abnormally long.
>
> Here is my code about PoolingHttpClientConnectionManager :
>
> cm = new PoolingHttpClientConnectionManager();
> cm.setMaxTotal(1024);
> cm.setDefaultMaxPerRoute(128);
>
>
>
> bit1...@163.com
>
> From: bit1...@163.com
> Date: 2015-07-08 10:25
> To: httpclient-users
> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
> HttpClient3
> Thanks Stefan for the reply.
>
> Do you mean that I move the CloseableHttpClient out of the try block like
> following,
>
> public String execute(HttpGet httpGet) {
> String body = "";
> CloseableHttpClient httpclient = this.getHttpclient();
> try{
> CloseableHttpResponse response = httpclient.execute(httpGet);
> int status = response.getStatusLine().getStatusCode();
> ///other codes goes for consuming response
>
> Unfortunately, It still doesn't work for me, and I still see that the
> request time is abnormally long.
>
> One thing I notice, is that when I set the socket time out to be 10
> seconds, there are couple of requests throw exception due to socket time
> out. So I wonder what may cause socket time out issue. With httpclient3, I
> see no such problems, and the longest request time is no more that 2
> seconds.
>
>
>
>
>
>
> bit1...@163.com
> From: Stefan Magnus Landrø
> Date: 2015-07-08 00:33
> To: HttpClient User Discussion
> Subject: Re: httpclient4 is extremely slow than correpsoning code of
> HttpClient3
> >> try{
> >> CloseableHttpClient httpclient =
> Try with resources will close client
> Sendt fra min iPhone
> > Den 7. jul. 2015 kl. 16.59 skrev Todd <bit1...@163.com>:
> >
> > Thanks Stefan for the reply. Could you please point to me which code you
> are referring to?
> >
> >
> >
> > At 2015-07-07 22:49:28, "Stefan Magnus Landrø" <stefan.lan...@gmail.com>
> wrote:
> >> You shouldn't close http client on every request, only response
> >>
> >> Sendt fra min iPhone
> >>
> >>> Den 7. jul. 2015 kl. 16.05 skrev "bit1...@163.com" <bit1...@163.com>:
> >>>
> >>>
> >>> Can someone kindly help me on this? Thanks a lot!
> >>>
> >>>
> >>>
> >>>
> >>> bit1...@163.com
> >>>
> >>> From: bit1...@163.com
> >>> Date: 2015-07-07 20:51
> >>> To: httpclient-users
> >>> Subject: httpclient4 is extremely slow than correpsoning code of
> HttpClient3
> >>>
> >>> I have following http client code(HttpUtils and HttpClientManager)with
> HttpClient 4.3.6, and a httpclient code in 3.1, and a test case as well. I
> observed that HttpClient 4.3.6 is very much slowly than HttpClient 3.1
> code. In HttpClient 4.3.6 version code, there are a lot
> >>> socket time out error(the socket time is 10 seconds).
> >>>
> >>> I pasted the related classes and code below. It is kind of long code,
> but I still would ask you do me a favor to review the http client
> configuration that may cause the problem. Many Thanks in advance.
> >>>
> >>> Following codes include:
> >>> 1. HttpClient 4.3.6 code to issue HttpGet request
> >>> 2. HttpClient 3.1 code to issue HttpGet request
> >>> 3. Test Case that demontrate the problem.
> >>>
> >>>
> >>> ############################################Http Client 4.3.6 Code
> goes here############################################
> >>>
> ///////////////////////////HttpUtils//////////////////////////////////////////
> >>> import java.util.ArrayList;
> >>> import java.util.List;
> >>> import java.util.Map;
> >>>
> >>> import org.apache.http.Consts;
> >>> import org.apache.http.NameValuePair;
> >>> import org.apache.http.client.config.RequestConfig;
> >>> import org.apache.http.client.entity.UrlEncodedFormEntity;
> >>> import org.apache.http.client.methods.HttpGet;
> >>> import org.apache.http.client.methods.HttpPost;
> >>> import org.apache.http.message.BasicNameValuePair;
> >>> import org.slf4j.Logger;
> >>> import org.slf4j.LoggerFactory;
> >>>
> >>>
> >>>
> >>> public class HttpUtils {
> >>>
> >>> ///This is the util method that will be used to issue http request.
> >>> public static String httpInvoke(String httpUrl, Map<String, Object>
> parameters, RequestConfig config){
> >>> HttpClientManager httpClientConnectionManager =
> HttpClientManager.getHttpClientConnectionManagerInstance();
> >>> HttpGet httpGet = new HttpGet(httpUrl);
> >>> if(config != null ){
> >>> httpGet.setConfig(config);
> >>> }
> >>> String result = httpClientConnectionManager.execute(httpGet);
> >>> return result;
> >>> }
> >>> }
> >>>
> >>>
> //////////////////////////////////HttpClientManager/////////////////////////////////
> >>>
> >>> import java.nio.charset.CodingErrorAction;
> >>> import java.util.ArrayList;
> >>> import java.util.Collection;
> >>>
> >>> import org.apache.commons.httpclient.Header;
> >>> import org.apache.commons.httpclient.HttpStatus;
> >>> import org.apache.commons.lang3.StringUtils;
> >>> import org.apache.http.Consts;
> >>> import org.apache.http.HttpEntity;
> >>> import org.apache.http.HttpHost;
> >>> import org.apache.http.client.config.RequestConfig;
> >>> import org.apache.http.client.entity.GzipDecompressingEntity;
> >>> import org.apache.http.client.methods.CloseableHttpResponse;
> >>> import org.apache.http.client.methods.HttpGet;
> >>> import org.apache.http.client.methods.HttpPost;
> >>> import org.apache.http.config.ConnectionConfig;
> >>> import org.apache.http.config.MessageConstraints;
> >>> import org.apache.http.config.SocketConfig;
> >>> import org.apache.http.impl.client.CloseableHttpClient;
> >>> import org.apache.http.impl.client.HttpClientBuilder;
> >>> import org.apache.http.impl.client.HttpClients;
> >>> import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
> >>> import org.apache.http.message.BasicHeader;
> >>> import org.apache.http.params.CoreProtocolPNames;
> >>> import org.apache.http.util.EntityUtils;
> >>> import org.slf4j.Logger;
> >>> import org.slf4j.LoggerFactory;
> >>>
> >>>
> >>> ///Main class that encapsulate the Http Client
> >>> public class HttpClientManager {
> >>> private static Logger logger =
> LoggerFactory.getLogger(HttpClientManager.class);
> >>>
> >>> private static int defaultConnectionTimeout = 10*1000; //connection
> timeout
> >>> private static int defaultSocketTimeout = 10*1000; //socket time out
> >>> private static int connectionRequestTimeout = 10*1000; //connection
> request timeout
> >>>
> >>> private static int defaultMaxRouteConnections = 128;
> >>> private static int defaultMaxTotalConnections = 1024;
> >>>
> >>> private static int defaultMaxHeaderCount = 200;
> >>> private static int defaultMaxLineLength = 2000;
> >>>
> >>> private static String Charset = "utf-8";
> >>>
> >>> private String proxyHost = null;
> >>> private String proxyPort =null;
> >>>
> >>> private PoolingHttpClientConnectionManager cm;
> >>>
> >>> private final static HttpClientManager httpClientConnectionManager =
> new HttpClientManager();
> >>>
> >>> private HttpClientManager() {
> >>> logger.info("HttpClientManager initial!");
> >>> cm = new PoolingHttpClientConnectionManager();
> >>> cm.setMaxTotal(defaultMaxTotalConnections);
> >>> cm.setDefaultMaxPerRoute(defaultMaxRouteConnections);
> >>> }
> >>>
> >>>
> >>> private CloseableHttpClient getHttpclient(){
> >>> HttpClientBuilder httpClientBuilder = HttpClients.custom();
> >>> httpClientBuilder.setConnectionManager(cm);
> >>>
> >>> RequestConfig.Builder requestConfigBuilder = RequestConfig.custom()
> >>> .setConnectTimeout(defaultConnectionTimeout)
> >>> .setSocketTimeout(defaultSocketTimeout)
> >>> .setConnectionRequestTimeout(connectionRequestTimeout)
> >>> .setExpectContinueEnabled(false)
> >>> .setStaleConnectionCheckEnabled(true);
> >>>
> >>> if(StringUtils.isNotBlank(proxyHost) &&
> StringUtils.isNotBlank(proxyPort)){
> >>> try{
> >>> logger.info("using proxy, proxyHost:{}, proxyPort:{}", proxyHost,
> proxyPort);
> >>> int proxyPortInt = Integer.parseInt(proxyPort);
> >>> requestConfigBuilder.setProxy(new HttpHost(proxyHost, proxyPortInt));
> >>> } catch(Exception e){
> >>> logger.error("parseInt proxyPort:{}", proxyPort, e);
> >>> }
> >>> }
> >>>
> >>> SocketConfig socketConfig =
> SocketConfig.custom().setTcpNoDelay(true).build();
> >>>
> >>> MessageConstraints messageConstraints =
> MessageConstraints.custom().setMaxHeaderCount(defaultMaxHeaderCount).setMaxLineLength(defaultMaxLineLength).build();
> >>>
> >>> ConnectionConfig connectionConfig = ConnectionConfig.custom()
> >>> .setMalformedInputAction(CodingErrorAction.IGNORE)
> >>> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> >>> .setCharset(Consts.UTF_8)
> >>> .setMessageConstraints(messageConstraints).build();
> >>>
> >>> Collection<BasicHeader> collection = new ArrayList<BasicHeader>();
> >>> collection.add(new BasicHeader("User-Agent", "Mozilla/5.0 (Windows; U;
> Windows NT 5.1; zh-CN; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3"));
> >>> collection.add(new BasicHeader("Accept-Language",
> "zh-cn,zh,en-US,en;q=0.5"));
> >>> collection.add(new BasicHeader("Accept-Charset", Charset));
> >>> collection.add(new BasicHeader("Accept-Encoding", "gzip"));
> >>>
> >>>
> httpClientBuilder.setDefaultRequestConfig(requestConfigBuilder.build());
> >>> httpClientBuilder.setDefaultSocketConfig(socketConfig);
> >>> httpClientBuilder.setDefaultConnectionConfig(connectionConfig);
> >>> httpClientBuilder.setDefaultHeaders(collection);
> >>>
> >>> return httpClientBuilder.build();
> >>> }
> >>>
> >>> ///This is the method that will be call the execute the HttpGet request
> >>> public String execute(HttpGet httpGet) {
> >>> String body = "";
> >>> try{
> >>> CloseableHttpClient httpclient = this.getHttpclient();
> >>> CloseableHttpResponse response = httpclient.execute(httpGet);
> >>> int status = response.getStatusLine().getStatusCode();
> >>> try {
> >>> if (status == HttpStatus.SC_OK) {
> >>> HttpEntity entity = response.getEntity();
> >>> if (entity != null) {
> >>> Header header = (Header) entity.getContentEncoding();
> >>> if(header != null && "gzip".equals(header.getValue())){
> >>> body = EntityUtils.toString(new GzipDecompressingEntity(entity),
> Charset);
> >>> } else {
> >>> body = EntityUtils.toString(entity, Charset);
> >>> }
> >>> }
> >>> } else {
> >>> logger.error("[httpClientManager] [fail] [httpGet:{}] [status:{}]",
> httpGet, status);
> >>> }
> >>> } finally {
> >>> response.close();
> >>> }
> >>> } catch(Exception e) {
> >>> logger.error("[module:httpClientManager] [action:execute] [httpGet:{}]
> [error:{}] ", httpGet, e.getMessage(), e);
> >>> }
> >>> return body;
> >>> }
> >>>
> >>>
> >>> ////Singleton object that will be used to access
> httpClientConnectionManager
> >>> public static HttpClientManager
> getHttpClientConnectionManagerInstance(){
> >>> return httpClientConnectionManager;
> >>> }
> >>>
> >>> }
> >>>
> >>>
> >>> ##################################################My HttpClient 1.3
> code goes here###########################################
> >>>
> >>> import org.apache.commons.httpclient.HttpClient;
> >>> import org.apache.commons.httpclient.HttpMethod;
> >>> import org.apache.commons.httpclient.methods.GetMethod;
> >>>
> >>> import java.io.BufferedReader;
> >>> import java.io.IOException;
> >>> import java.io.InputStream;
> >>> import java.io.InputStreamReader;
> >>>
> >>> public class HttpClientUtil {
> >>>
> >>> private static final HttpClientUtil INSTANCE = new HttpClientUtil();
> >>>
> >>> private static final int TIMEOUT = 10 * 1000; //10s
> >>>
> >>> private HttpClientUtil() {
> >>> }
> >>>
> >>> ///Actually, this class is more like a static class than Singleton
> >>> public static HttpClientUtil getInstance() {
> >>> return INSTANCE;
> >>> }
> >>>
> >>> //brand new http client per request
> >>> private HttpClient newHttpClient() {
> >>> HttpClient client = new HttpClient();
> >>>
> client.getHttpConnectionManager().getParams().setConnectionTimeout(TIMEOUT);
> >>> client.getHttpConnectionManager().getParams().setSoTimeout(TIMEOUT);
> >>> return client;
> >>> }
> >>>
> >>> //FIXME The encoding should be provided when convert the response
> binary stream into string
> >>> public static String responseBodyAsString(HttpMethod method) throws
> IOException {
> >>> BufferedReader br = null;
> >>> String lsr = System.getProperty("line.separator");
> >>> try {
> >>> InputStream in = method.getResponseBodyAsStream();
> >>> br = new BufferedReader(new InputStreamReader(in, "UTF-8"));
> >>> StringBuffer sb = new StringBuffer();
> >>> String line;
> >>> while ((line = br.readLine()) != null) {
> >>> sb.append(line).append(lsr);
> >>> }
> >>> return sb.toString();
> >>> } finally {
> >>> if (br != null) {
> >>> br.close();
> >>> }
> >>> }
> >>> }
> >>>
> >>> /////This is the method that will HttpGet to the url
> >>> public String get(String url) throws IOException {
> >>> GetMethod pm = new GetMethod(url);
> >>> pm.setRequestHeader("Connection", "close");
> >>> HttpClient client = newHttpClient();
> >>> try {
> >>> client.executeMethod(pm);
> >>> String response = responseBodyAsString(pm);
> >>> return response;
> >>> } finally {
> >>> pm.releaseConnection();
> >>> }
> >>> }
> >>> }
> >>>
> >>> ##########################################Test
> Case##########################################
> >>> The following code runs in the one thread in the junit test
> >>> while (i++ < 5000) {
> >>> long start = System.currentTimeMillis();
> >>> HttpUtils.httpInvoke(TEST_URL, null, null);
> >>> long a = System.currentTimeMillis() - start;
> >>> String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
> >>> writeToFile(timeSpent + "\n", logFile);
> >>> Thread.sleep(loopInterval);
> >>> }
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> bit1...@163.com
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >>
>
