On Wed, 2015-05-27 at 15:42 +0200, Thomas Boniface wrote: > Olged when you say "You really do not want several dozen threads trying to > initiate a request all at the same time.". I'm a bit surprised this > basically what our application need to do and what I though http async was > about. This leads me to the following questions: > > Why shouldn't we do this
Because this is causing lock contention you have been seeing. There are various ways of making sure that does not happen but they all imply certain design decisions. > or should we use http-async to be compliant with > its design ? Should we look into another technology if http-async does not > fit this use case ? > One generally uses non-blocking i/o in order to have few threads manage many connections. In your case though you have quite a number of threads managing incoming connections and the same number of outgoing connections. If all those threads race to execute a request on an outgoing connection, you get what you get: contention on the lock that guards the connection pool. So, you have to decide if that is what you want. If it is not, you have options: (1) build your own lock-less pool at the expense of not being able to impose a strict max total / max per route limit. (2) use an intermediate queue to queue requests and use a small number of threads to submit them to the client for execution at the expense of higher complexity. Oleg PS: generally, building a good HTTP proxy / gateway is _hard_. > Thanks > > 2015-05-27 13:26 GMT+02:00 Oleg Kalnichevski <[email protected]>: > > > On Wed, 2015-05-27 at 10:51 +0200, Thomas Boniface wrote: > > > Hi, > > > > > > We managed to observe our issue once again, this time with connection > > > logging. Here is the situation: > > > > > > No activity was detected on the application log for a few seconds, > > > triggering a tomcat thread dump. > > > > > > The thread dump shows all http-nio-127.0.0.1-8080-exec-* threads waiting > > on > > > > > org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection > > > while all I/O dispatcher threads are waiting on > > > > > org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseConnection. > > > > > > The application log (covering the last 2 sec of activity before the > > > application being stuck) shows activity on the http async client until a > > > point where we received requests that are supposed to trigger http > > > communication to upstream servers but nothing is logged by http client (I > > > assume because the threads from incoming requests are waiting to lease a > > > connection). > > > > > > Here are the corresponding logs: > > > http://s000.tinyupload.com/index.php?file_id=15495126975418045294 > > > > > > Thanks, > > > Thomas > > > > > > > Thomas > > I am sorry but really cannot sift through 300MB of logs. What I can > > glean from a cursory look is there appears to be no dead-lock, just a > > lot of threads contending for the lock <0x00000000be315a50>. The lock > > does not look to be locked by any tread. > > > > Therefore this is unlikely to be due to a bug in HttpClient and more > > likely to be due to the design of your application. One thing that I > > find bizarre is that why there are so many threads are contending for > > the lock in the first place. This looks wrong. You really do not want > > several dozen threads trying to initiate a request all at the same time. > > > > Oleg > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
