AW: Why are many TCP open connections created on OS (Linux)?
Hello Bernd, thank you for your reply. I see them as ESTABLISHED and TIME_WAIT in netstat. The connections with ESTABLISHED state are ok and most under 100, but the connections with TIME_WAIT state are always more than 200. I mean on the client. On the server is ok and has little open connections. Gruss ;) Manuel Here some important code, how I set it up: public class NetworkClient implements AutoCloseable { ... public NetworkClient() { ConnectingIOReactor ioReactor; try { // Create I/O reactor configuration IOReactorConfig ioReactorConfig = IOReactorConfig.custom() .setIoThreadCount(Runtime.getRuntime().availableProcessors()) .setConnectTimeout(config.getConnectTimeout()).setSoTimeout(config.getSoTimeout()) .build(); ioReactor = new DefaultConnectingIOReactor(ioReactorConfig); PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor); cm.setMaxTotal(config.getConnectionMaxTotal()); cm.setDefaultMaxPerRoute(config.getConnectionDefaultMaxPerRoute()); RequestConfig globalConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.IGNORE_COOKIES) .build(); HttpProcessor httpprocessor = HttpProcessorBuilder.create().build(); httpClient = HttpAsyncClients.custom().setDefaultRequestConfig(globalConfig) .setHttpProcessor(httpprocessor).setConnectionManager(cm).build(); } catch (IOReactorException e) { LOGGER.error("Could not create ConnectionIOReactor", e); } initClientTransport(); } private void initClientTransport() { httpClient.start(); } public Integer send(String destination, int port, HttpRequest httpRequest, String operationName) { Assert.notNull(destination); Assert.notNull(httpRequest); startTime = startTime == 0 ? System.nanoTime() : startTime; HttpHost target = new HttpHost(destination, port); NcResponse ncResponse = getNewNcResponse(operationName); NcResponseProducer responseProducer = new NcResponseProducer(target, httpRequest, ncResponse, requestCounter); NcResponseConsumer responseConsumer = new NcResponseConsumer(ncResponse); httpClient.execute(responseProducer, responseConsumer, new ClientCallback(responseCounter, failCounter, connectionErrorMessageList, getRequestLine(httpRequest))); long clientId = requestClientId.getAndIncrement(); return ((Long) clientId).intValue(); } .. } public class NcResponseProducer extends BasicAsyncRequestProducer { private final NcResponse ncResponse; private static final Logger LOGGER = LogManager.getLogger(); private final AtomicLong requestCounter; private final Header[] allHeaders; public NcResponseProducer(HttpHost target, HttpRequest request, NcResponse ncResponse, AtomicLong requestCounter) { super(target, request); this.allHeaders = request.getAllHeaders(); this.ncResponse = ncResponse; this.requestCounter = requestCounter; } @Override public void requestCompleted(HttpContext context) { super.requestCompleted(context); ncResponse.setStartTimeInNano(System.nanoTime()); LOGGER.debug("Sent request to client {}. request ID={} headers={} count={}", getTarget().toHostString(), ncResponse.getRequestID(), Arrays.asList(allHeaders), requestCounter.incrementAndGet()); } } public class NcResponseConsumer extends BasicAsyncResponseConsumer { private final NcResponse ncResponse; public NcResponseConsumer(NcResponse ncResponse) { this.ncResponse = ncResponse; } @Override protected void onResponseReceived(HttpResponse response) throws IOException { super.onResponseReceived(response); ncResponse.setEndTimeInNano(System.nanoTime()); ncResponse.setReturnCode(response.getStatusLine().getStatusCode()); } } -Ursprüngliche Nachricht- Von: Bernd Eckenfels [mailto:e...@zusammenkunft.net] Gesendet: Dienstag, 4. Oktober 2016 21:19 An: Gnerlich, Manuel (IPS PROJECTS GMBH) Cc: httpclient-users@hc.apache.org Betreff: Re: Why are many TCP open connections created on OS (Linux)? Am Tue, 4 Oct 2016 14:00:38 + schrieb "Gnerlich, Manuel (IPS PROJECTS GMBH)" : > I am using Apache AsychHttpClient with connection pool to send more > than 1 mio requests and configured maxinum number of connection (200) > and of connection per Host (100). > > However there are many open TCP connections (about 30.000) on linux. > Why? How can I control number of connection over Java on linux? Define open for us, please. Do you mean you see them as ESTABLISHED in netstat or do you mean you see them as socket handles for the Java process (in lsof). Do you mean on the client or the server? Can you show some code how you set it up and use the client. Gruss Bernd - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional command
Re: AW: Why are many TCP open connections created on OS (Linux)?
On Wed, 2016-10-05 at 12:16 +, Gnerlich, Manuel (IPS PROJECTS GMBH) wrote: > Hello Bernd, > > thank you for your reply. > > I see them as ESTABLISHED and TIME_WAIT in netstat. The connections with > ESTABLISHED state are ok and most under 100, but the connections with > TIME_WAIT state are always more than 200. I mean on the client. On the server > is ok and has little open connections. Please take a look at this resource: http://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions Oleg > > Gruss ;) Manuel > > Here some important code, how I set it up: > > public class NetworkClient implements AutoCloseable { > ... > public NetworkClient() { > ConnectingIOReactor ioReactor; > try { > // Create I/O reactor configuration > IOReactorConfig ioReactorConfig = IOReactorConfig.custom() > .setIoThreadCount(Runtime.getRuntime().availableProcessors()) > > .setConnectTimeout(config.getConnectTimeout()).setSoTimeout(config.getSoTimeout()) > .build(); > ioReactor = new DefaultConnectingIOReactor(ioReactorConfig); > > PoolingNHttpClientConnectionManager cm = new > PoolingNHttpClientConnectionManager(ioReactor); > cm.setMaxTotal(config.getConnectionMaxTotal()); > cm.setDefaultMaxPerRoute(config.getConnectionDefaultMaxPerRoute()); > > RequestConfig globalConfig = > RequestConfig.custom().setCookieSpec(CookieSpecs.IGNORE_COOKIES) > .build(); > > HttpProcessor httpprocessor = HttpProcessorBuilder.create().build(); > httpClient = > HttpAsyncClients.custom().setDefaultRequestConfig(globalConfig) > .setHttpProcessor(httpprocessor).setConnectionManager(cm).build(); > } catch (IOReactorException e) { > LOGGER.error("Could not create ConnectionIOReactor", e); > } > > initClientTransport(); > } > > private void initClientTransport() { > httpClient.start(); > } > > > > public Integer send(String destination, int port, HttpRequest httpRequest, > String operationName) { > Assert.notNull(destination); > Assert.notNull(httpRequest); > > startTime = startTime == 0 ? System.nanoTime() : startTime; > > HttpHost target = new HttpHost(destination, port); > > NcResponse ncResponse = getNewNcResponse(operationName); > > NcResponseProducer responseProducer = new NcResponseProducer(target, > httpRequest, ncResponse, > requestCounter); > NcResponseConsumer responseConsumer = new NcResponseConsumer(ncResponse); > > httpClient.execute(responseProducer, responseConsumer, new > ClientCallback(responseCounter, > failCounter, connectionErrorMessageList, > getRequestLine(httpRequest))); > > long clientId = requestClientId.getAndIncrement(); > return ((Long) clientId).intValue(); > } > .. > } > > public class NcResponseProducer extends BasicAsyncRequestProducer { > > private final NcResponse ncResponse; > > private static final Logger LOGGER = LogManager.getLogger(); > > private final AtomicLong requestCounter; > > private final Header[] allHeaders; > > public NcResponseProducer(HttpHost target, HttpRequest request, NcResponse > ncResponse, > AtomicLong requestCounter) { > super(target, request); > this.allHeaders = request.getAllHeaders(); > this.ncResponse = ncResponse; > this.requestCounter = requestCounter; > } > > @Override > public void requestCompleted(HttpContext context) { > super.requestCompleted(context); > ncResponse.setStartTimeInNano(System.nanoTime()); > LOGGER.debug("Sent request to client {}. request ID={} headers={} > count={}", > getTarget().toHostString(), ncResponse.getRequestID(), > Arrays.asList(allHeaders), > requestCounter.incrementAndGet()); > } > > } > > public class NcResponseConsumer extends BasicAsyncResponseConsumer { > > private final NcResponse ncResponse; > > public NcResponseConsumer(NcResponse ncResponse) { > this.ncResponse = ncResponse; > } > > @Override > protected void onResponseReceived(HttpResponse response) throws IOException > { > super.onResponseReceived(response); > ncResponse.setEndTimeInNano(System.nanoTime()); > ncResponse.setReturnCode(response.getStatusLine().getStatusCode()); > } > > } > > > -Ursprüngliche Nachricht- > Von: Bernd Eckenfels [mailto:e...@zusammenkunft.net] > Gesendet: Dienstag, 4. Oktober 2016 21:19 > An: Gnerlich, Manuel (IPS PROJECTS GMBH) > Cc: httpclient-users@hc.apache.org > Betreff: Re: Why are many TCP open connections created on OS (Linux)? > > Am Tue, 4 Oct 2016 14:00:38 + > schrieb "Gnerlich, Manuel (IPS PROJECTS GMBH)" > : > > > I am using Apache AsychHttpClient with connection pool to send more > > than 1 mio requests and configured maxinum number of connection (200) > > and of connection per Host (100). > > > > However there are many open TCP connections (about 30.000) on linux.