Slow connection and timeouts when using localhost on macOS with Java / HTTP when network disconnected

[Christopher BROWN]

Hello,

I've noticed (as of today, when running a demo for a customer) that, with
Java 8 (u121) on macOS Sierra, when both wi-fi is disconnected and no
ethernet is available, Java network performance is slow.  The problem
occurs when I have one application (Tomcat) being called by another (Java,
using HTTP Client 4.3.x to connect to Tomcat, and also embedding a Jetty
server, on a different port to Tomcat).  Both applications connect to a
PostgreSQL database using the TCP/IP-based JDBC driver, where the database
is in a VirtualBox VM on the same machine, using NAT to access it locally
on port 5432.

Starting Tomcat (bound to localhost:8080) is slow, as is starting the
embedded Jetty server.  I suspect this is either Java or the MacOS
networking stack being inefficient about resolving "localhost" when there's
no network or DNS (but, I also tried using 127.0.0.1 instead of localhost,
and didn't get any improvement).  Both eventually bind (after about 15
seconds) to server sockets then respond promptly when say accessing content
with a local web browser.  But Java calls to localhost:8080 are so slow,
that they just time out (I could I suppose set a really long timeout, but
performance would be totally unacceptable).

I don't think it's HTTP client's fault, but I can't think of how to set
things up for acceptable performance.  For what it's worth, I've copied a
stacktrace below, but I'm really looking for advice about how to improve
things (I didn't see anything helpful in Java's networking properties, and
the fact that 127.0.0.1 didn't help is odd).  Reconnecting to a network
instantly solves the problem, but that's not always an option (when I
develop during travel, for example).

Any advice ?  Thanks in advance.

java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:87)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at vsh.system.connector.NetworkDriver.ping(NetworkDriver.java:1081)

--
Christopher

Re: I/O thread blocking under high concurrent load

[Sudheera Palihakkara]

Hi,

Yes of course. The reason behind my concern is all these blocks seems to be
occurred in EPoll related operations and there are quite a lot of these
blocks. I'm worried about this has a impact on the performance of the
application. Is there any configuration parameter to switch off this EPoll
usage in order to perform a round of testing without it. Thanks

Regards,
Sudheera.

On Thu, Feb 2, 2017 at 5:36 PM, Oleg Kalnichevski  wrote:

> On February 2, 2017 7:05:07 AM GMT+01:00, Sudheera Palihakkara <
> catchsudhe...@gmail.com> wrote:
> >Hi all,
> >
> >I'm currently using httpcore-nio 4.4 for a proxy-like application and
> >make
> >use of the file channels when content transferring. Recently I have
> >profiled my code under a high concurrent load and observed few unusual
> >thread blocking occurrences. Stack traces are listed below.
> >
> >*I/O threads*
> >
> >S-I/O-dispatcher-1 [BLOCKED]
> >sun.nio.ch.EPollSelectorImpl.doSelect(long) EPollSelectorImpl.java:88
> >java.lang.Thread.run() Thread.java:745
> >
> >S-I/O-dispatcher-5 [BLOCKED]
> >org.apache.http.impl.nio.reactor.IOSessionImpl.setEvent(int)
> >IOSessionImpl.java:164
> >org.apache.http.impl.nio.DefaultNHttpClientConnection.
> submitRequest(HttpRequest)
> >DefaultNHttpClientConnection.java:339
> >
> >*Worker threds*
> >
> >pool-2-thread-99 [BLOCKED]
> >sun.nio.ch.EPollArrayWrapper.setInterest(int, int)
> >EPollArrayWrapper.java:213
> >org.apache.http.nio.protocol.HttpAsyncService$HttpAsyncExchangeImpl.
> submitResponse(HttpAsyncResponseProducer)
> >HttpAsyncService.java:1037
> >
> >pool-2-thread-96 [BLOCKED]
> >sun.nio.ch.EPollSelectorImpl.wakeup() EPollSelectorImpl.java:191
> >org.apache.http.nio.protocol.HttpAsyncService$HttpAsyncExchangeImpl.
> submitResponse(HttpAsyncResponseProducer)
> >HttpAsyncService.java:1037
> >
> >Previously I have used httpcore-nio 4.2.5 for the same application and
> >did
> >not encountered any such thread blockings. Can you help me out to
> >figure
> >out what might be the issue here? Thank you very much.
>
> Synchronized sections of code can cause threads to block on the
> synchronization object under load. This does not necessarily mean there is
> a problem.
>
> Oleg
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>



-- 
*Sudheera Palihakkara.*
Undergraduate
Department of
*Computer Science and Engineering,*Faculty of Engineering,
*University of Moratuwa*,
Sri Lanka.

Re: I/O thread blocking under high concurrent load

[Oleg Kalnichevski]

On February 2, 2017 7:05:07 AM GMT+01:00, Sudheera Palihakkara 
 wrote:
>Hi all,
>
>I'm currently using httpcore-nio 4.4 for a proxy-like application and
>make
>use of the file channels when content transferring. Recently I have
>profiled my code under a high concurrent load and observed few unusual
>thread blocking occurrences. Stack traces are listed below.
>
>*I/O threads*
>
>S-I/O-dispatcher-1 [BLOCKED]
>sun.nio.ch.EPollSelectorImpl.doSelect(long) EPollSelectorImpl.java:88
>java.lang.Thread.run() Thread.java:745
>
>S-I/O-dispatcher-5 [BLOCKED]
>org.apache.http.impl.nio.reactor.IOSessionImpl.setEvent(int)
>IOSessionImpl.java:164
>org.apache.http.impl.nio.DefaultNHttpClientConnection.submitRequest(HttpRequest)
>DefaultNHttpClientConnection.java:339
>
>*Worker threds*
>
>pool-2-thread-99 [BLOCKED]
>sun.nio.ch.EPollArrayWrapper.setInterest(int, int)
>EPollArrayWrapper.java:213
>org.apache.http.nio.protocol.HttpAsyncService$HttpAsyncExchangeImpl.submitResponse(HttpAsyncResponseProducer)
>HttpAsyncService.java:1037
>
>pool-2-thread-96 [BLOCKED]
>sun.nio.ch.EPollSelectorImpl.wakeup() EPollSelectorImpl.java:191
>org.apache.http.nio.protocol.HttpAsyncService$HttpAsyncExchangeImpl.submitResponse(HttpAsyncResponseProducer)
>HttpAsyncService.java:1037
>
>Previously I have used httpcore-nio 4.2.5 for the same application and
>did
>not encountered any such thread blockings. Can you help me out to
>figure
>out what might be the issue here? Thank you very much.

Synchronized sections of code can cause threads to block on the synchronization 
object under load. This does not necessarily mean there is a problem.

Oleg
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org