I would suggest going further and creating 2 buildable projects with maven poms. So one can easily run it. On Jul 8, 2015 11:57 AM, "bit1...@163.com" <bit1...@163.com> wrote:
> Ok, thanks Stefan! > > The github repo is https://github.com/bit1129/bit-repo . That are 5 java > files there with the discription (Http Client 3.x and Http Client 4.3.6 > problematic code) > > Thanks very much for you guys's time on this!! > > > > > > bit1...@163.com > > From: Stefan Magnus Landrø > Date: 2015-07-08 12:58 > To: HttpClient User Discussion > Subject: Re: httpclient4 is extremely slow than correpsoning code of > HttpClient3 > Can you create a github repo with your code? Reviewing gets easier then > > Sendt fra min iPhone > > > Den 8. jul. 2015 kl. 06.03 skrev "bit1...@163.com" <bit1...@163.com>: > > > > Thanks Alexey for the apply。 I tried it, and sorry that it still doesn't > work. I am using one thread to sending http request for 1000 times every 20 > milliseconds. Some response time are as follows: > > As you can see, the response time is not stable at all, (there are > ocurrences with 7~10 seconds). If I don't use pool connection manager, then > the response time are very stable, about 200 milliseconds for each request. > > > > Is there code example that issuing HTTP request with pool connection > manager and apply the best practices for the CloseableHttpClient/response > release/close or related things. So that, I can compare it with mine, and > hopefully find whether the problem is. > > Thanks. > > > > My testing code snippet: > > > > public void test() { > > long allStart = System.currentTimeMillis(); > > while (i++ < loop) { > > long start = System.currentTimeMillis(); > > HttpUtils.httpInvoke(TEST_URL, null, null); //This the core code that > issuing http request. > > long a = System.currentTimeMillis() - start; > > String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a); > > writeToFile(timeSpent + "\n", logFile); > > Thread.sleep(loopInterval); > > } > > writeToFile("Total time spent: " + (System.currentTimeMillis() - > allStart) + "\n", logFile); > > > > > > } > > > > 26 > > 30 > > 235 > > 439 > > 851 > > 1667 > > 1040 > > 234 > > 439 > > 851 > > 1671 > > 3310 > > 25 > > 27 > > 26 > > 238 > > 25 > > 234 > > 435 > > 24 > > 235 > > 24 > > 26 > > 27 > > 25 > > 232 > > 27 > > 26 > > 37 > > 29 > > 28 > > 251 > > 472 > > 915 > > 1804 > > 3580 > > 7132 > > 10057 > > 28 > > 26 > > 253 > > 26 > > 25 > > 23 > > 243 > > 24 > > 24 > > 27 > > 26 > > 1025 > > 27 > > 29 > > 24 > > 23 > > > > > > > > > > bit1...@163.com > > > > From: Alexey Panchenko > > Date: 2015-07-08 11:42 > > To: HttpClient User Discussion > > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of > HttpClient3 > > AFAIK there is no need in creating a new HttpClient instance every time, > it > > should be created once and reused. > > > >> On Wed, Jul 8, 2015 at 9:29 AM, bit1...@163.com <bit1...@163.com> > wrote: > >> > >> One more thing that I noticed is: > >> > >> When I constructing the CloseableHttpClient, response time gets back to > >> normal if I don't pass the PoolingHttpClientConnectionManager object to > the > >> HttpClientBuilder > >> > >> HttpClientBuilder httpClientBuilder = HttpClients.custom(); > >> > >> /*comment this code so that no PoolingHttpClientConnectionManager will > >> be used!*/ > >> //httpClientBuilder.setConnectionManager(cm); > >> > >> //other configurations > >> return httpClientBuilder.build(); > >> > >> Seems that there are something related with > >> PoolingHttpClientConnectionManager that cause the request's response > time > >> is abnormally long. > >> > >> Here is my code about PoolingHttpClientConnectionManager : > >> > >> cm = new PoolingHttpClientConnectionManager(); > >> cm.setMaxTotal(1024); > >> cm.setDefaultMaxPerRoute(128); > >> > >> > >> > >> bit1...@163.com > >> > >> From: bit1...@163.com > >> Date: 2015-07-08 10:25 > >> To: httpclient-users > >> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of > >> HttpClient3 > >> Thanks Stefan for the reply. > >> > >> Do you mean that I move the CloseableHttpClient out of the try block > like > >> following, > >> > >> public String execute(HttpGet httpGet) { > >> String body = ""; > >> CloseableHttpClient httpclient = this.getHttpclient(); > >> try{ > >> CloseableHttpResponse response = httpclient.execute(httpGet); > >> int status = response.getStatusLine().getStatusCode(); > >> ///other codes goes for consuming response > >> > >> Unfortunately, It still doesn't work for me, and I still see that the > >> request time is abnormally long. > >> > >> One thing I notice, is that when I set the socket time out to be 10 > >> seconds, there are couple of requests throw exception due to socket time > >> out. So I wonder what may cause socket time out issue. With > httpclient3, I > >> see no such problems, and the longest request time is no more that 2 > >> seconds. > >> > >> > >> > >> > >> > >> > >> bit1...@163.com > >> From: Stefan Magnus Landrø > >> Date: 2015-07-08 00:33 > >> To: HttpClient User Discussion > >> Subject: Re: httpclient4 is extremely slow than correpsoning code of > >> HttpClient3 > >>>> try{ > >>>> CloseableHttpClient httpclient = > >> Try with resources will close client > >> Sendt fra min iPhone > >>> Den 7. jul. 2015 kl. 16.59 skrev Todd <bit1...@163.com>: > >>> > >>> Thanks Stefan for the reply. Could you please point to me which code > you > >> are referring to? > >>> > >>> > >>> > >>> At 2015-07-07 22:49:28, "Stefan Magnus Landrø" < > stefan.lan...@gmail.com> > >> wrote: > >>>> You shouldn't close http client on every request, only response > >>>> > >>>> Sendt fra min iPhone > >>>> > >>>>> Den 7. jul. 2015 kl. 16.05 skrev "bit1...@163.com" <bit1...@163.com > >: > >>>>> > >>>>> > >>>>> Can someone kindly help me on this? Thanks a lot! > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> bit1...@163.com > >>>>> > >>>>> From: bit1...@163.com > >>>>> Date: 2015-07-07 20:51 > >>>>> To: httpclient-users > >>>>> Subject: httpclient4 is extremely slow than correpsoning code of > >> HttpClient3 > >>>>> > >>>>> I have following http client code(HttpUtils and > HttpClientManager)with > >> HttpClient 4.3.6, and a httpclient code in 3.1, and a test case as > well. I > >> observed that HttpClient 4.3.6 is very much slowly than HttpClient 3.1 > >> code. In HttpClient 4.3.6 version code, there are a lot > >>>>> socket time out error(the socket time is 10 seconds). > >>>>> > >>>>> I pasted the related classes and code below. It is kind of long code, > >> but I still would ask you do me a favor to review the http client > >> configuration that may cause the problem. Many Thanks in advance. > >>>>> > >>>>> Following codes include: > >>>>> 1. HttpClient 4.3.6 code to issue HttpGet request > >>>>> 2. HttpClient 3.1 code to issue HttpGet request > >>>>> 3. Test Case that demontrate the problem. > >>>>> > >>>>> > >>>>> ############################################Http Client 4.3.6 Code > >> goes here############################################ > >> > ///////////////////////////HttpUtils////////////////////////////////////////// > >>>>> import java.util.ArrayList; > >>>>> import java.util.List; > >>>>> import java.util.Map; > >>>>> > >>>>> import org.apache.http.Consts; > >>>>> import org.apache.http.NameValuePair; > >>>>> import org.apache.http.client.config.RequestConfig; > >>>>> import org.apache.http.client.entity.UrlEncodedFormEntity; > >>>>> import org.apache.http.client.methods.HttpGet; > >>>>> import org.apache.http.client.methods.HttpPost; > >>>>> import org.apache.http.message.BasicNameValuePair; > >>>>> import org.slf4j.Logger; > >>>>> import org.slf4j.LoggerFactory; > >>>>> > >>>>> > >>>>> > >>>>> public class HttpUtils { > >>>>> > >>>>> ///This is the util method that will be used to issue http request. > >>>>> public static String httpInvoke(String httpUrl, Map<String, Object> > >> parameters, RequestConfig config){ > >>>>> HttpClientManager httpClientConnectionManager = > >> HttpClientManager.getHttpClientConnectionManagerInstance(); > >>>>> HttpGet httpGet = new HttpGet(httpUrl); > >>>>> if(config != null ){ > >>>>> httpGet.setConfig(config); > >>>>> } > >>>>> String result = httpClientConnectionManager.execute(httpGet); > >>>>> return result; > >>>>> } > >>>>> } > >> > //////////////////////////////////HttpClientManager///////////////////////////////// > >>>>> > >>>>> import java.nio.charset.CodingErrorAction; > >>>>> import java.util.ArrayList; > >>>>> import java.util.Collection; > >>>>> > >>>>> import org.apache.commons.httpclient.Header; > >>>>> import org.apache.commons.httpclient.HttpStatus; > >>>>> import org.apache.commons.lang3.StringUtils; > >>>>> import org.apache.http.Consts; > >>>>> import org.apache.http.HttpEntity; > >>>>> import org.apache.http.HttpHost; > >>>>> import org.apache.http.client.config.RequestConfig; > >>>>> import org.apache.http.client.entity.GzipDecompressingEntity; > >>>>> import org.apache.http.client.methods.CloseableHttpResponse; > >>>>> import org.apache.http.client.methods.HttpGet; > >>>>> import org.apache.http.client.methods.HttpPost; > >>>>> import org.apache.http.config.ConnectionConfig; > >>>>> import org.apache.http.config.MessageConstraints; > >>>>> import org.apache.http.config.SocketConfig; > >>>>> import org.apache.http.impl.client.CloseableHttpClient; > >>>>> import org.apache.http.impl.client.HttpClientBuilder; > >>>>> import org.apache.http.impl.client.HttpClients; > >>>>> import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; > >>>>> import org.apache.http.message.BasicHeader; > >>>>> import org.apache.http.params.CoreProtocolPNames; > >>>>> import org.apache.http.util.EntityUtils; > >>>>> import org.slf4j.Logger; > >>>>> import org.slf4j.LoggerFactory; > >>>>> > >>>>> > >>>>> ///Main class that encapsulate the Http Client > >>>>> public class HttpClientManager { > >>>>> private static Logger logger = > >> LoggerFactory.getLogger(HttpClientManager.class); > >>>>> > >>>>> private static int defaultConnectionTimeout = 10*1000; //connection > >> timeout > >>>>> private static int defaultSocketTimeout = 10*1000; //socket time out > >>>>> private static int connectionRequestTimeout = 10*1000; //connection > >> request timeout > >>>>> > >>>>> private static int defaultMaxRouteConnections = 128; > >>>>> private static int defaultMaxTotalConnections = 1024; > >>>>> > >>>>> private static int defaultMaxHeaderCount = 200; > >>>>> private static int defaultMaxLineLength = 2000; > >>>>> > >>>>> private static String Charset = "utf-8"; > >>>>> > >>>>> private String proxyHost = null; > >>>>> private String proxyPort =null; > >>>>> > >>>>> private PoolingHttpClientConnectionManager cm; > >>>>> > >>>>> private final static HttpClientManager httpClientConnectionManager = > >> new HttpClientManager(); > >>>>> > >>>>> private HttpClientManager() { > >>>>> logger.info("HttpClientManager initial!"); > >>>>> cm = new PoolingHttpClientConnectionManager(); > >>>>> cm.setMaxTotal(defaultMaxTotalConnections); > >>>>> cm.setDefaultMaxPerRoute(defaultMaxRouteConnections); > >>>>> } > >>>>> > >>>>> > >>>>> private CloseableHttpClient getHttpclient(){ > >>>>> HttpClientBuilder httpClientBuilder = HttpClients.custom(); > >>>>> httpClientBuilder.setConnectionManager(cm); > >>>>> > >>>>> RequestConfig.Builder requestConfigBuilder = RequestConfig.custom() > >>>>> .setConnectTimeout(defaultConnectionTimeout) > >>>>> .setSocketTimeout(defaultSocketTimeout) > >>>>> .setConnectionRequestTimeout(connectionRequestTimeout) > >>>>> .setExpectContinueEnabled(false) > >>>>> .setStaleConnectionCheckEnabled(true); > >>>>> > >>>>> if(StringUtils.isNotBlank(proxyHost) && > >> StringUtils.isNotBlank(proxyPort)){ > >>>>> try{ > >>>>> logger.info("using proxy, proxyHost:{}, proxyPort:{}", proxyHost, > >> proxyPort); > >>>>> int proxyPortInt = Integer.parseInt(proxyPort); > >>>>> requestConfigBuilder.setProxy(new HttpHost(proxyHost, proxyPortInt)); > >>>>> } catch(Exception e){ > >>>>> logger.error("parseInt proxyPort:{}", proxyPort, e); > >>>>> } > >>>>> } > >>>>> > >>>>> SocketConfig socketConfig = > >> SocketConfig.custom().setTcpNoDelay(true).build(); > >>>>> > >>>>> MessageConstraints messageConstraints = > >> > MessageConstraints.custom().setMaxHeaderCount(defaultMaxHeaderCount).setMaxLineLength(defaultMaxLineLength).build(); > >>>>> > >>>>> ConnectionConfig connectionConfig = ConnectionConfig.custom() > >>>>> .setMalformedInputAction(CodingErrorAction.IGNORE) > >>>>> .setUnmappableInputAction(CodingErrorAction.IGNORE) > >>>>> .setCharset(Consts.UTF_8) > >>>>> .setMessageConstraints(messageConstraints).build(); > >>>>> > >>>>> Collection<BasicHeader> collection = new ArrayList<BasicHeader>(); > >>>>> collection.add(new BasicHeader("User-Agent", "Mozilla/5.0 (Windows; > U; > >> Windows NT 5.1; zh-CN; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3")); > >>>>> collection.add(new BasicHeader("Accept-Language", > >> "zh-cn,zh,en-US,en;q=0.5")); > >>>>> collection.add(new BasicHeader("Accept-Charset", Charset)); > >>>>> collection.add(new BasicHeader("Accept-Encoding", "gzip")); > >> httpClientBuilder.setDefaultRequestConfig(requestConfigBuilder.build()); > >>>>> httpClientBuilder.setDefaultSocketConfig(socketConfig); > >>>>> httpClientBuilder.setDefaultConnectionConfig(connectionConfig); > >>>>> httpClientBuilder.setDefaultHeaders(collection); > >>>>> > >>>>> return httpClientBuilder.build(); > >>>>> } > >>>>> > >>>>> ///This is the method that will be call the execute the HttpGet > request > >>>>> public String execute(HttpGet httpGet) { > >>>>> String body = ""; > >>>>> try{ > >>>>> CloseableHttpClient httpclient = this.getHttpclient(); > >>>>> CloseableHttpResponse response = httpclient.execute(httpGet); > >>>>> int status = response.getStatusLine().getStatusCode(); > >>>>> try { > >>>>> if (status == HttpStatus.SC_OK) { > >>>>> HttpEntity entity = response.getEntity(); > >>>>> if (entity != null) { > >>>>> Header header = (Header) entity.getContentEncoding(); > >>>>> if(header != null && "gzip".equals(header.getValue())){ > >>>>> body = EntityUtils.toString(new GzipDecompressingEntity(entity), > >> Charset); > >>>>> } else { > >>>>> body = EntityUtils.toString(entity, Charset); > >>>>> } > >>>>> } > >>>>> } else { > >>>>> logger.error("[httpClientManager] [fail] [httpGet:{}] [status:{}]", > >> httpGet, status); > >>>>> } > >>>>> } finally { > >>>>> response.close(); > >>>>> } > >>>>> } catch(Exception e) { > >>>>> logger.error("[module:httpClientManager] [action:execute] > [httpGet:{}] > >> [error:{}] ", httpGet, e.getMessage(), e); > >>>>> } > >>>>> return body; > >>>>> } > >>>>> > >>>>> > >>>>> ////Singleton object that will be used to access > >> httpClientConnectionManager > >>>>> public static HttpClientManager > >> getHttpClientConnectionManagerInstance(){ > >>>>> return httpClientConnectionManager; > >>>>> } > >>>>> > >>>>> } > >>>>> > >>>>> > >>>>> ##################################################My HttpClient 1.3 > >> code goes here########################################### > >>>>> > >>>>> import org.apache.commons.httpclient.HttpClient; > >>>>> import org.apache.commons.httpclient.HttpMethod; > >>>>> import org.apache.commons.httpclient.methods.GetMethod; > >>>>> > >>>>> import java.io.BufferedReader; > >>>>> import java.io.IOException; > >>>>> import java.io.InputStream; > >>>>> import java.io.InputStreamReader; > >>>>> > >>>>> public class HttpClientUtil { > >>>>> > >>>>> private static final HttpClientUtil INSTANCE = new HttpClientUtil(); > >>>>> > >>>>> private static final int TIMEOUT = 10 * 1000; //10s > >>>>> > >>>>> private HttpClientUtil() { > >>>>> } > >>>>> > >>>>> ///Actually, this class is more like a static class than Singleton > >>>>> public static HttpClientUtil getInstance() { > >>>>> return INSTANCE; > >>>>> } > >>>>> > >>>>> //brand new http client per request > >>>>> private HttpClient newHttpClient() { > >>>>> HttpClient client = new HttpClient(); > >> > client.getHttpConnectionManager().getParams().setConnectionTimeout(TIMEOUT); > >>>>> client.getHttpConnectionManager().getParams().setSoTimeout(TIMEOUT); > >>>>> return client; > >>>>> } > >>>>> > >>>>> //FIXME The encoding should be provided when convert the response > >> binary stream into string > >>>>> public static String responseBodyAsString(HttpMethod method) throws > >> IOException { > >>>>> BufferedReader br = null; > >>>>> String lsr = System.getProperty("line.separator"); > >>>>> try { > >>>>> InputStream in = method.getResponseBodyAsStream(); > >>>>> br = new BufferedReader(new InputStreamReader(in, "UTF-8")); > >>>>> StringBuffer sb = new StringBuffer(); > >>>>> String line; > >>>>> while ((line = br.readLine()) != null) { > >>>>> sb.append(line).append(lsr); > >>>>> } > >>>>> return sb.toString(); > >>>>> } finally { > >>>>> if (br != null) { > >>>>> br.close(); > >>>>> } > >>>>> } > >>>>> } > >>>>> > >>>>> /////This is the method that will HttpGet to the url > >>>>> public String get(String url) throws IOException { > >>>>> GetMethod pm = new GetMethod(url); > >>>>> pm.setRequestHeader("Connection", "close"); > >>>>> HttpClient client = newHttpClient(); > >>>>> try { > >>>>> client.executeMethod(pm); > >>>>> String response = responseBodyAsString(pm); > >>>>> return response; > >>>>> } finally { > >>>>> pm.releaseConnection(); > >>>>> } > >>>>> } > >>>>> } > >>>>> > >>>>> ##########################################Test > >> Case########################################## > >>>>> The following code runs in the one thread in the junit test > >>>>> while (i++ < 5000) { > >>>>> long start = System.currentTimeMillis(); > >>>>> HttpUtils.httpInvoke(TEST_URL, null, null); > >>>>> long a = System.currentTimeMillis() - start; > >>>>> String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a); > >>>>> writeToFile(timeSpent + "\n", logFile); > >>>>> Thread.sleep(loopInterval); > >>>>> } > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> bit1...@163.com > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > >>>> For additional commands, e-mail: httpclient-users-h...@hc.apache.org > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > >
