Few things here:
HttpClientPoolUtilsTest:
why not use a logging framework instead (log4j, logback, commons logging
etc) of creating log files manually?
Why all this static code?
HttpClientManager Line 122: this willl invoke getHttpclient that will build
a brand new client on every time. Most inefficient.
You should instead use a singleton pattern:
public class HttpUtil {
private CloseableHttpClient client = // ... build it in constructor
public String execute(String url) {
client.execute( ... )
}
}
Maybe
https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html is
a better fit for you?
2015-07-08 8:49 GMT+02:00 bit1...@163.com <bit1...@163.com>:
> Thanks Alexey for your suggestion.
> I have created a maven project and it is here :
> https://github.com/bit1129/bit-repo/tree/master/HttpClientUsage
>
> Thanks!
>
>
>
> bit1...@163.com
>
> From: Alexey Panchenko
> Date: 2015-07-08 14:05
> To: HttpClient User Discussion
> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
> HttpClient3
> I would suggest going further and creating 2 buildable projects with maven
> poms. So one can easily run it.
> On Jul 8, 2015 11:57 AM, "bit1...@163.com" <bit1...@163.com> wrote:
>
> > Ok, thanks Stefan!
> >
> > The github repo is https://github.com/bit1129/bit-repo . That are 5 java
> > files there with the discription (Http Client 3.x and Http Client 4.3.6
> > problematic code)
> >
> > Thanks very much for you guys's time on this!!
> >
> >
> >
> >
> >
> > bit1...@163.com
> >
> > From: Stefan Magnus Landrø
> > Date: 2015-07-08 12:58
> > To: HttpClient User Discussion
> > Subject: Re: httpclient4 is extremely slow than correpsoning code of
> > HttpClient3
> > Can you create a github repo with your code? Reviewing gets easier then
> >
> > Sendt fra min iPhone
> >
> > > Den 8. jul. 2015 kl. 06.03 skrev "bit1...@163.com" <bit1...@163.com>:
> > >
> > > Thanks Alexey for the apply。 I tried it, and sorry that it still
> doesn't
> > work. I am using one thread to sending http request for 1000 times every
> 20
> > milliseconds. Some response time are as follows:
> > > As you can see, the response time is not stable at all, (there are
> > ocurrences with 7~10 seconds). If I don't use pool connection manager,
> then
> > the response time are very stable, about 200 milliseconds for each
> request.
> > >
> > > Is there code example that issuing HTTP request with pool connection
> > manager and apply the best practices for the CloseableHttpClient/response
> > release/close or related things. So that, I can compare it with mine, and
> > hopefully find whether the problem is.
> > > Thanks.
> > >
> > > My testing code snippet:
> > >
> > > public void test() {
> > > long allStart = System.currentTimeMillis();
> > > while (i++ < loop) {
> > > long start = System.currentTimeMillis();
> > > HttpUtils.httpInvoke(TEST_URL, null, null); //This the core code that
> > issuing http request.
> > > long a = System.currentTimeMillis() - start;
> > > String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
> > > writeToFile(timeSpent + "\n", logFile);
> > > Thread.sleep(loopInterval);
> > > }
> > > writeToFile("Total time spent: " + (System.currentTimeMillis() -
> > allStart) + "\n", logFile);
> > >
> > >
> > > }
> > >
> > > 26
> > > 30
> > > 235
> > > 439
> > > 851
> > > 1667
> > > 1040
> > > 234
> > > 439
> > > 851
> > > 1671
> > > 3310
> > > 25
> > > 27
> > > 26
> > > 238
> > > 25
> > > 234
> > > 435
> > > 24
> > > 235
> > > 24
> > > 26
> > > 27
> > > 25
> > > 232
> > > 27
> > > 26
> > > 37
> > > 29
> > > 28
> > > 251
> > > 472
> > > 915
> > > 1804
> > > 3580
> > > 7132
> > > 10057
> > > 28
> > > 26
> > > 253
> > > 26
> > > 25
> > > 23
> > > 243
> > > 24
> > > 24
> > > 27
> > > 26
> > > 1025
> > > 27
> > > 29
> > > 24
> > > 23
> > >
> > >
> > >
> > >
> > > bit1...@163.com
> > >
> > > From: Alexey Panchenko
> > > Date: 2015-07-08 11:42
> > > To: HttpClient User Discussion
> > > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code
> of
> > HttpClient3
> > > AFAIK there is no need in creating a new HttpClient instance every
> time,
> > it
> > > should be created once and reused.
> > >
> > >> On Wed, Jul 8, 2015 at 9:29 AM, bit1...@163.com <bit1...@163.com>
> > wrote:
> > >>
> > >> One more thing that I noticed is:
> > >>
> > >> When I constructing the CloseableHttpClient, response time gets back
> to
> > >> normal if I don't pass the PoolingHttpClientConnectionManager object
> to
> > the
> > >> HttpClientBuilder
> > >>
> > >> HttpClientBuilder httpClientBuilder = HttpClients.custom();
> > >>
> > >> /*comment this code so that no PoolingHttpClientConnectionManager
> will
> > >> be used!*/
> > >> //httpClientBuilder.setConnectionManager(cm);
> > >>
> > >> //other configurations
> > >> return httpClientBuilder.build();
> > >>
> > >> Seems that there are something related with
> > >> PoolingHttpClientConnectionManager that cause the request's response
> > time
> > >> is abnormally long.
> > >>
> > >> Here is my code about PoolingHttpClientConnectionManager :
> > >>
> > >> cm = new PoolingHttpClientConnectionManager();
> > >> cm.setMaxTotal(1024);
> > >> cm.setDefaultMaxPerRoute(128);
> > >>
> > >>
> > >>
> > >> bit1...@163.com
> > >>
> > >> From: bit1...@163.com
> > >> Date: 2015-07-08 10:25
> > >> To: httpclient-users
> > >> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code
> of
> > >> HttpClient3
> > >> Thanks Stefan for the reply.
> > >>
> > >> Do you mean that I move the CloseableHttpClient out of the try block
> > like
> > >> following,
> > >>
> > >> public String execute(HttpGet httpGet) {
> > >> String body = "";
> > >> CloseableHttpClient httpclient = this.getHttpclient();
> > >> try{
> > >> CloseableHttpResponse response = httpclient.execute(httpGet);
> > >> int status = response.getStatusLine().getStatusCode();
> > >> ///other codes goes for consuming response
> > >>
> > >> Unfortunately, It still doesn't work for me, and I still see that the
> > >> request time is abnormally long.
> > >>
> > >> One thing I notice, is that when I set the socket time out to be 10
> > >> seconds, there are couple of requests throw exception due to socket
> time
> > >> out. So I wonder what may cause socket time out issue. With
> > httpclient3, I
> > >> see no such problems, and the longest request time is no more that 2
> > >> seconds.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> bit1...@163.com
> > >> From: Stefan Magnus Landrø
> > >> Date: 2015-07-08 00:33
> > >> To: HttpClient User Discussion
> > >> Subject: Re: httpclient4 is extremely slow than correpsoning code of
> > >> HttpClient3
> > >>>> try{
> > >>>> CloseableHttpClient httpclient =
> > >> Try with resources will close client
> > >> Sendt fra min iPhone
> > >>> Den 7. jul. 2015 kl. 16.59 skrev Todd <bit1...@163.com>:
> > >>>
> > >>> Thanks Stefan for the reply. Could you please point to me which code
> > you
> > >> are referring to?
> > >>>
> > >>>
> > >>>
> > >>> At 2015-07-07 22:49:28, "Stefan Magnus Landrø" <
> > stefan.lan...@gmail.com>
> > >> wrote:
> > >>>> You shouldn't close http client on every request, only response
> > >>>>
> > >>>> Sendt fra min iPhone
> > >>>>
> > >>>>> Den 7. jul. 2015 kl. 16.05 skrev "bit1...@163.com" <
> bit1...@163.com
> > >:
> > >>>>>
> > >>>>>
> > >>>>> Can someone kindly help me on this? Thanks a lot!
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> bit1...@163.com
> > >>>>>
> > >>>>> From: bit1...@163.com
> > >>>>> Date: 2015-07-07 20:51
> > >>>>> To: httpclient-users
> > >>>>> Subject: httpclient4 is extremely slow than correpsoning code of
> > >> HttpClient3
> > >>>>>
> > >>>>> I have following http client code(HttpUtils and
> > HttpClientManager)with
> > >> HttpClient 4.3.6, and a httpclient code in 3.1, and a test case as
> > well. I
> > >> observed that HttpClient 4.3.6 is very much slowly than HttpClient 3.1
> > >> code. In HttpClient 4.3.6 version code, there are a lot
> > >>>>> socket time out error(the socket time is 10 seconds).
> > >>>>>
> > >>>>> I pasted the related classes and code below. It is kind of long
> code,
> > >> but I still would ask you do me a favor to review the http client
> > >> configuration that may cause the problem. Many Thanks in advance.
> > >>>>>
> > >>>>> Following codes include:
> > >>>>> 1. HttpClient 4.3.6 code to issue HttpGet request
> > >>>>> 2. HttpClient 3.1 code to issue HttpGet request
> > >>>>> 3. Test Case that demontrate the problem.
> > >>>>>
> > >>>>>
> > >>>>> ############################################Http Client 4.3.6 Code
> > >> goes here############################################
> > >>
> >
> ///////////////////////////HttpUtils//////////////////////////////////////////
> > >>>>> import java.util.ArrayList;
> > >>>>> import java.util.List;
> > >>>>> import java.util.Map;
> > >>>>>
> > >>>>> import org.apache.http.Consts;
> > >>>>> import org.apache.http.NameValuePair;
> > >>>>> import org.apache.http.client.config.RequestConfig;
> > >>>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
> > >>>>> import org.apache.http.client.methods.HttpGet;
> > >>>>> import org.apache.http.client.methods.HttpPost;
> > >>>>> import org.apache.http.message.BasicNameValuePair;
> > >>>>> import org.slf4j.Logger;
> > >>>>> import org.slf4j.LoggerFactory;
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> public class HttpUtils {
> > >>>>>
> > >>>>> ///This is the util method that will be used to issue http request.
> > >>>>> public static String httpInvoke(String httpUrl, Map<String, Object>
> > >> parameters, RequestConfig config){
> > >>>>> HttpClientManager httpClientConnectionManager =
> > >> HttpClientManager.getHttpClientConnectionManagerInstance();
> > >>>>> HttpGet httpGet = new HttpGet(httpUrl);
> > >>>>> if(config != null ){
> > >>>>> httpGet.setConfig(config);
> > >>>>> }
> > >>>>> String result = httpClientConnectionManager.execute(httpGet);
> > >>>>> return result;
> > >>>>> }
> > >>>>> }
> > >>
> >
> //////////////////////////////////HttpClientManager/////////////////////////////////
> > >>>>>
> > >>>>> import java.nio.charset.CodingErrorAction;
> > >>>>> import java.util.ArrayList;
> > >>>>> import java.util.Collection;
> > >>>>>
> > >>>>> import org.apache.commons.httpclient.Header;
> > >>>>> import org.apache.commons.httpclient.HttpStatus;
> > >>>>> import org.apache.commons.lang3.StringUtils;
> > >>>>> import org.apache.http.Consts;
> > >>>>> import org.apache.http.HttpEntity;
> > >>>>> import org.apache.http.HttpHost;
> > >>>>> import org.apache.http.client.config.RequestConfig;
> > >>>>> import org.apache.http.client.entity.GzipDecompressingEntity;
> > >>>>> import org.apache.http.client.methods.CloseableHttpResponse;
> > >>>>> import org.apache.http.client.methods.HttpGet;
> > >>>>> import org.apache.http.client.methods.HttpPost;
> > >>>>> import org.apache.http.config.ConnectionConfig;
> > >>>>> import org.apache.http.config.MessageConstraints;
> > >>>>> import org.apache.http.config.SocketConfig;
> > >>>>> import org.apache.http.impl.client.CloseableHttpClient;
> > >>>>> import org.apache.http.impl.client.HttpClientBuilder;
> > >>>>> import org.apache.http.impl.client.HttpClients;
> > >>>>> import
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
> > >>>>> import org.apache.http.message.BasicHeader;
> > >>>>> import org.apache.http.params.CoreProtocolPNames;
> > >>>>> import org.apache.http.util.EntityUtils;
> > >>>>> import org.slf4j.Logger;
> > >>>>> import org.slf4j.LoggerFactory;
> > >>>>>
> > >>>>>
> > >>>>> ///Main class that encapsulate the Http Client
> > >>>>> public class HttpClientManager {
> > >>>>> private static Logger logger =
> > >> LoggerFactory.getLogger(HttpClientManager.class);
> > >>>>>
> > >>>>> private static int defaultConnectionTimeout = 10*1000;
> //connection
> > >> timeout
> > >>>>> private static int defaultSocketTimeout = 10*1000; //socket time
> out
> > >>>>> private static int connectionRequestTimeout = 10*1000;
> //connection
> > >> request timeout
> > >>>>>
> > >>>>> private static int defaultMaxRouteConnections = 128;
> > >>>>> private static int defaultMaxTotalConnections = 1024;
> > >>>>>
> > >>>>> private static int defaultMaxHeaderCount = 200;
> > >>>>> private static int defaultMaxLineLength = 2000;
> > >>>>>
> > >>>>> private static String Charset = "utf-8";
> > >>>>>
> > >>>>> private String proxyHost = null;
> > >>>>> private String proxyPort =null;
> > >>>>>
> > >>>>> private PoolingHttpClientConnectionManager cm;
> > >>>>>
> > >>>>> private final static HttpClientManager httpClientConnectionManager
> =
> > >> new HttpClientManager();
> > >>>>>
> > >>>>> private HttpClientManager() {
> > >>>>> logger.info("HttpClientManager initial!");
> > >>>>> cm = new PoolingHttpClientConnectionManager();
> > >>>>> cm.setMaxTotal(defaultMaxTotalConnections);
> > >>>>> cm.setDefaultMaxPerRoute(defaultMaxRouteConnections);
> > >>>>> }
> > >>>>>
> > >>>>>
> > >>>>> private CloseableHttpClient getHttpclient(){
> > >>>>> HttpClientBuilder httpClientBuilder = HttpClients.custom();
> > >>>>> httpClientBuilder.setConnectionManager(cm);
> > >>>>>
> > >>>>> RequestConfig.Builder requestConfigBuilder = RequestConfig.custom()
> > >>>>> .setConnectTimeout(defaultConnectionTimeout)
> > >>>>> .setSocketTimeout(defaultSocketTimeout)
> > >>>>> .setConnectionRequestTimeout(connectionRequestTimeout)
> > >>>>> .setExpectContinueEnabled(false)
> > >>>>> .setStaleConnectionCheckEnabled(true);
> > >>>>>
> > >>>>> if(StringUtils.isNotBlank(proxyHost) &&
> > >> StringUtils.isNotBlank(proxyPort)){
> > >>>>> try{
> > >>>>> logger.info("using proxy, proxyHost:{}, proxyPort:{}", proxyHost,
> > >> proxyPort);
> > >>>>> int proxyPortInt = Integer.parseInt(proxyPort);
> > >>>>> requestConfigBuilder.setProxy(new HttpHost(proxyHost,
> proxyPortInt));
> > >>>>> } catch(Exception e){
> > >>>>> logger.error("parseInt proxyPort:{}", proxyPort, e);
> > >>>>> }
> > >>>>> }
> > >>>>>
> > >>>>> SocketConfig socketConfig =
> > >> SocketConfig.custom().setTcpNoDelay(true).build();
> > >>>>>
> > >>>>> MessageConstraints messageConstraints =
> > >>
> >
> MessageConstraints.custom().setMaxHeaderCount(defaultMaxHeaderCount).setMaxLineLength(defaultMaxLineLength).build();
> > >>>>>
> > >>>>> ConnectionConfig connectionConfig = ConnectionConfig.custom()
> > >>>>> .setMalformedInputAction(CodingErrorAction.IGNORE)
> > >>>>> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> > >>>>> .setCharset(Consts.UTF_8)
> > >>>>> .setMessageConstraints(messageConstraints).build();
> > >>>>>
> > >>>>> Collection<BasicHeader> collection = new ArrayList<BasicHeader>();
> > >>>>> collection.add(new BasicHeader("User-Agent", "Mozilla/5.0 (Windows;
> > U;
> > >> Windows NT 5.1; zh-CN; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3"));
> > >>>>> collection.add(new BasicHeader("Accept-Language",
> > >> "zh-cn,zh,en-US,en;q=0.5"));
> > >>>>> collection.add(new BasicHeader("Accept-Charset", Charset));
> > >>>>> collection.add(new BasicHeader("Accept-Encoding", "gzip"));
> > >>
> httpClientBuilder.setDefaultRequestConfig(requestConfigBuilder.build());
> > >>>>> httpClientBuilder.setDefaultSocketConfig(socketConfig);
> > >>>>> httpClientBuilder.setDefaultConnectionConfig(connectionConfig);
> > >>>>> httpClientBuilder.setDefaultHeaders(collection);
> > >>>>>
> > >>>>> return httpClientBuilder.build();
> > >>>>> }
> > >>>>>
> > >>>>> ///This is the method that will be call the execute the HttpGet
> > request
> > >>>>> public String execute(HttpGet httpGet) {
> > >>>>> String body = "";
> > >>>>> try{
> > >>>>> CloseableHttpClient httpclient = this.getHttpclient();
> > >>>>> CloseableHttpResponse response = httpclient.execute(httpGet);
> > >>>>> int status = response.getStatusLine().getStatusCode();
> > >>>>> try {
> > >>>>> if (status == HttpStatus.SC_OK) {
> > >>>>> HttpEntity entity = response.getEntity();
> > >>>>> if (entity != null) {
> > >>>>> Header header = (Header) entity.getContentEncoding();
> > >>>>> if(header != null && "gzip".equals(header.getValue())){
> > >>>>> body = EntityUtils.toString(new GzipDecompressingEntity(entity),
> > >> Charset);
> > >>>>> } else {
> > >>>>> body = EntityUtils.toString(entity, Charset);
> > >>>>> }
> > >>>>> }
> > >>>>> } else {
> > >>>>> logger.error("[httpClientManager] [fail] [httpGet:{}] [status:{}]",
> > >> httpGet, status);
> > >>>>> }
> > >>>>> } finally {
> > >>>>> response.close();
> > >>>>> }
> > >>>>> } catch(Exception e) {
> > >>>>> logger.error("[module:httpClientManager] [action:execute]
> > [httpGet:{}]
> > >> [error:{}] ", httpGet, e.getMessage(), e);
> > >>>>> }
> > >>>>> return body;
> > >>>>> }
> > >>>>>
> > >>>>>
> > >>>>> ////Singleton object that will be used to access
> > >> httpClientConnectionManager
> > >>>>> public static HttpClientManager
> > >> getHttpClientConnectionManagerInstance(){
> > >>>>> return httpClientConnectionManager;
> > >>>>> }
> > >>>>>
> > >>>>> }
> > >>>>>
> > >>>>>
> > >>>>> ##################################################My HttpClient 1.3
> > >> code goes here###########################################
> > >>>>>
> > >>>>> import org.apache.commons.httpclient.HttpClient;
> > >>>>> import org.apache.commons.httpclient.HttpMethod;
> > >>>>> import org.apache.commons.httpclient.methods.GetMethod;
> > >>>>>
> > >>>>> import java.io.BufferedReader;
> > >>>>> import java.io.IOException;
> > >>>>> import java.io.InputStream;
> > >>>>> import java.io.InputStreamReader;
> > >>>>>
> > >>>>> public class HttpClientUtil {
> > >>>>>
> > >>>>> private static final HttpClientUtil INSTANCE = new
> HttpClientUtil();
> > >>>>>
> > >>>>> private static final int TIMEOUT = 10 * 1000; //10s
> > >>>>>
> > >>>>> private HttpClientUtil() {
> > >>>>> }
> > >>>>>
> > >>>>> ///Actually, this class is more like a static class than Singleton
> > >>>>> public static HttpClientUtil getInstance() {
> > >>>>> return INSTANCE;
> > >>>>> }
> > >>>>>
> > >>>>> //brand new http client per request
> > >>>>> private HttpClient newHttpClient() {
> > >>>>> HttpClient client = new HttpClient();
> > >>
> >
> client.getHttpConnectionManager().getParams().setConnectionTimeout(TIMEOUT);
> > >>>>>
> client.getHttpConnectionManager().getParams().setSoTimeout(TIMEOUT);
> > >>>>> return client;
> > >>>>> }
> > >>>>>
> > >>>>> //FIXME The encoding should be provided when convert the response
> > >> binary stream into string
> > >>>>> public static String responseBodyAsString(HttpMethod method) throws
> > >> IOException {
> > >>>>> BufferedReader br = null;
> > >>>>> String lsr = System.getProperty("line.separator");
> > >>>>> try {
> > >>>>> InputStream in = method.getResponseBodyAsStream();
> > >>>>> br = new BufferedReader(new InputStreamReader(in, "UTF-8"));
> > >>>>> StringBuffer sb = new StringBuffer();
> > >>>>> String line;
> > >>>>> while ((line = br.readLine()) != null) {
> > >>>>> sb.append(line).append(lsr);
> > >>>>> }
> > >>>>> return sb.toString();
> > >>>>> } finally {
> > >>>>> if (br != null) {
> > >>>>> br.close();
> > >>>>> }
> > >>>>> }
> > >>>>> }
> > >>>>>
> > >>>>> /////This is the method that will HttpGet to the url
> > >>>>> public String get(String url) throws IOException {
> > >>>>> GetMethod pm = new GetMethod(url);
> > >>>>> pm.setRequestHeader("Connection", "close");
> > >>>>> HttpClient client = newHttpClient();
> > >>>>> try {
> > >>>>> client.executeMethod(pm);
> > >>>>> String response = responseBodyAsString(pm);
> > >>>>> return response;
> > >>>>> } finally {
> > >>>>> pm.releaseConnection();
> > >>>>> }
> > >>>>> }
> > >>>>> }
> > >>>>>
> > >>>>> ##########################################Test
> > >> Case##########################################
> > >>>>> The following code runs in the one thread in the junit test
> > >>>>> while (i++ < 5000) {
> > >>>>> long start = System.currentTimeMillis();
> > >>>>> HttpUtils.httpInvoke(TEST_URL, null, null);
> > >>>>> long a = System.currentTimeMillis() - start;
> > >>>>> String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
> > >>>>> writeToFile(timeSpent + "\n", logFile);
> > >>>>> Thread.sleep(loopInterval);
> > >>>>> }
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> bit1...@163.com
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > >>>> For additional commands, e-mail:
> httpclient-users-h...@hc.apache.org
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> >
>
--
BEKK Open
http://open.bekk.no
TesTcl - a unit test framework for iRules
http://testcl.com
