I have a project that need to send 1000 httpRequest concurrently from to client
to server.
First i use HttpClient's FutureRequestExecutionService to send the request. But
found that send 1000 httpRequest concurrently will consume so long time.
So i want to use HttpAsyncClients to decrease the time.
unfortunately i found that HttpAsyncClients need more time to send 1000
httpRequest concurrently.
Why the asynchronous way take more to time to complete? please help.
Here is my test code. I send 100 request to www.baidu.com, and print the time
it take to response. The length of the response is 52999 bytes.
HttpAsyncClients's test code:
import java.util.concurrent.atomic.AtomicInteger; import
javax.net.ssl.SSLContext; import org.apache.http.HttpHost; import
org.apache.http.HttpResponse; import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpHead; import
org.apache.http.concurrent.FutureCallback; import
org.apache.http.config.Registry; import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.ssl.BrowserCompatHostnameVerifier; import
org.apache.http.conn.ssl.SSLContexts; import
org.apache.http.conn.ssl.X509HostnameVerifier; import
org.apache.http.impl.conn.DefaultProxyRoutePlanner; import
org.apache.http.impl.nio.client.CloseableHttpAsyncClient; import
org.apache.http.impl.nio.client.HttpAsyncClients; import
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager; import
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor; import
org.apache.http.impl.nio.reactor.IOReactorConfig; import
org.apache.http.nio.conn.NoopIOSessionStrategy; import
org.apache.http.nio.conn.SchemeIOSessionStrategy; import
org.apache.http.nio.conn.ssl.SSLIOSessionStrategy; import
org.apache.http.nio.reactor.ConnectingIOReactor; import
com.yygame.gscanner.ConstList; public class HttpAsyncClientTest { public
static void main(String[] args) { try { SSLContext
sslcontext = SSLContexts.createSystemDefault();
X509HostnameVerifier hostnameVerifier = new BrowserCompatHostnameVerifier();
Registry<SchemeIOSessionStrategy> sessionStrategyRegistry =
RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", new SSLIOSessionStrategy(sslcontext, hostnameVerifier))
.build(); CloseableHttpAsyncClient httpClient;
IOReactorConfig ioReactorConfig = IOReactorConfig
.custom() .setIoThreadCount(12)
.build(); ConnectingIOReactor ioReactor = new
DefaultConnectingIOReactor(ioReactorConfig);
PoolingNHttpClientConnectionManager connManager = new
PoolingNHttpClientConnectionManager( ioReactor, null,
sessionStrategyRegistry, null); connManager.setMaxTotal(100);
connManager.setDefaultMaxPerRoute(100); httpClient =
HttpAsyncClients.custom()
.setConnectionManager(connManager) .build();
httpClient.start(); String[] hosts = new String[] {
"http://www.baidu.com"; }; final AtomicInteger count = new
AtomicInteger(0); final long startTime =
System.currentTimeMillis(); for (int i = 0; i < 100; ++i) {
final HttpGet request = new HttpGet(hosts[0]);
httpClient.execute(request, new
FutureCallback<HttpResponse>() { @Override
public void cancelled() { //
TODO Auto-generated method stub }
@Override public void
completed(HttpResponse arg0) { // TODO
Auto-generated method stub
System.out.println(count.addAndGet(1) + ".response: threadId="
+ Thread.currentThread().getId()
+ ", delay time=" +
(System.currentTimeMillis() - startTime)
+ "milisecond"); }
@Override public void failed(Exception arg0) {
// TODO Auto-generated method stub
} }); } } catch
(Exception e) { e.printStackTrace(); } } }
the test result is:
1.response: threadId=13, delay time=1030 (milisecond) 2.response: threadId=21,
delay time=1199 (milisecond) 3.response: threadId=10, delay time=1254
(milisecond) 4.response: threadId=16, delay time=1269 (milisecond) 5.response:
threadId=10, delay time=1301 (milisecond) 6.response: threadId=21, delay
time=1387 (milisecond) ..... 96.response: threadId=11, delay time=5912
(milisecond) 97.response: threadId=12, delay time=7214 (milisecond)
98.response: threadId=20, delay time=7234 (milisecond) 99.response:
threadId=10, delay time=7284 (milisecond)
FutureRequestExecutionService's test code:
import java.io.IOException; import java.util.ArrayList; import
java.util.concurrent.ExecutorService; import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger; import
org.apache.http.HttpHost; import org.apache.http.HttpResponse; import
org.apache.http.client.ClientProtocolException; import
org.apache.http.client.HttpClient; import
org.apache.http.client.ResponseHandler; import
org.apache.http.client.methods.HttpGet; import
org.apache.http.client.protocol.HttpClientContext; import
org.apache.http.concurrent.FutureCallback; import
org.apache.http.impl.client.FutureRequestExecutionService; import
org.apache.http.impl.client.HttpClientBuilder; import
org.apache.http.impl.client.HttpRequestFutureTask; import
org.apache.http.impl.conn.DefaultProxyRoutePlanner; public class
HttpClientFutureRequestTest { public static void main(String[] args) {
HttpClient httpClient = HttpClientBuilder.create()
.setMaxConnPerRoute(10) .build(); ExecutorService
executorService = Executors.newFixedThreadPool(12);
FutureRequestExecutionService futureRequestExecutionService =
new FutureRequestExecutionService(httpClient, executorService); final
AtomicInteger count = new AtomicInteger(0); final long startTime =
System.currentTimeMillis();
ArrayList<HttpRequestFutureTask<HttpResponse>> tasks = new
ArrayList<HttpRequestFutureTask<HttpResponse>>(); for(int i = 0; i <
100; ++i) { tasks.add(futureRequestExecutionService.execute(
new HttpGet("http://www.baidu.com";),
HttpClientContext.create(), new MyHandler(),
new FutureCallback<HttpResponse>() { @Override
public void cancelled() { //
TODO Auto-generated method stub }
@Override public void completed(HttpResponse arg0) {
// TODO Auto-generated method stub
System.out.println(count.addAndGet(1) + ".response: threadId="
+ Thread.currentThread().getId()
+ ", delay time=" +
(System.currentTimeMillis() - startTime) +
"milisecond"); } @Override
public void failed(Exception arg0) {
// TODO Auto-generated method stub }
})); } } private final static class MyHandler implements
ResponseHandler<HttpResponse> { @Override public
HttpResponse handleResponse(HttpResponse arg0) throws ClientProtocolException,
IOException { return arg0; } } }
the test result is:
1.response: threadId=10, delay time=607 (milisecond) 2.response: threadId=14,
delay time=613 (milisecond) 3.response: threadId=15, delay time=613
(milisecond) 4.response: threadId=12, delay time=618 (milisecond) 5.response:
threadId=9, delay time=623 (milisecond) 6.response: threadId=16, delay time=623
(milisecond) 96.response: threadId=14, delay time=3684 (milisecond)
97.response: threadId=9, delay time=3694 (milisecond) 98.response: threadId=11,
delay time=3774 (milisecond) 99.response: threadId=16, delay time=3788
(milisecond)
