[ https://issues.apache.org/jira/browse/HTTPASYNC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085520#comment-15085520 ]
Oleg Kalnichevski commented on HTTPASYNC-99: -------------------------------------------- I am unable to reproduce the problem on my Debian 8.2. Could you please capture the stack trace of the exception that kills the thread used by CloseableHttpAsyncClientBase? {noformat} Creating sockets Making request Socket exception: Address already in use Wait for a response Failed Closing sockets Closing client Client closed Exception in thread "main" java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:70) at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:98) at abnormalioreactor.AbnormalIOReactor.getResponse(AbnormalIOReactor.java:80) at abnormalioreactor.AbnormalIOReactor.main(AbnormalIOReactor.java:103) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:350) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) at java.lang.Thread.run(Thread.java:745) {noformat} > Response handling problems when too many open files as request executed > ----------------------------------------------------------------------- > > Key: HTTPASYNC-99 > URL: https://issues.apache.org/jira/browse/HTTPASYNC-99 > Project: HttpComponents HttpAsyncClient > Issue Type: Bug > Affects Versions: 4.1.1 > Reporter: Jeremy Barlow > Assignee: Oleg Kalnichevski > Attachments: AbnormalIOReactor.java, ResponseCallback.java > > > We have encountered a few issues with handling an HttpAsyncClient request > which is made when the process has too many open file descriptors: > 1) A .get() call made on the future returned from the httpclient's .execute() > method hangs indefinitely. > 2) No call is made to any of the methods on the > {{FutureCallback<HttpResponse>}} object provided as an argument to the > httpclient's .execute() method. I was expecting a call to .failed() to occur. > 3) A call to the httpclient's .close() method does not error out but also > does not result in all of the IOReactor "I/O Dispatcher" threads being shut > down. In normal cases, these threads would get shut down when the .close() > call is made. > I've attached a couple of Java source files which demonstrate the problem - > AbnormalIOReactor.java and ResponseCallback.java. AbnormalIOReactor.java is > based on the example at > https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchange.java. > Just after the httpclient is started, this class allocates bare sockets > repeatedly and holds onto them. This is done to force the application to get > into the state where file descriptors have been exhausted. The > httpclient.execute() call does not throw an error. > The class repeatedly calls .get() on the future response with a prolonged > timeout, breaking out of the loop when the client moves from the state of > having .isRunning() be true to .isRunning() being false. Checking for a > change in the .isRunning() state is a bit contrived here, ideally something > that an API client would not have to do. I noticed from the implementation > of the {{CloseableHttpAsyncClientBase}} constructor at > https://svn.apache.org/repos/asf/httpcomponents/httpasyncclient/tags/4.1.1/httpasyncclient/src/main/java/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java > that the reactor thread's run() method will catch an exception at request > startup and set the client status to {{STOPPED}} and this seemed like the > only way that a client could detect that a problem had occurred in this > situation. > {code:java} > @Override > public void run() { > try { > final IOEventDispatch ioEventDispatch = new > InternalIODispatch(handler); > connmgr.execute(ioEventDispatch); > } catch (final Exception ex) { > log.error("I/O reactor terminated abnormally", ex); > } finally { > status.set(Status.STOPPED); > } > } > {code} > The implementation of ResponseCallback.java writes a message to stdout when > any of the methods is invoked. In this case, no message is written to stdout > even though the request has effectively failed. > When this program is run from the command line like {{java > abnormalioreactor.AbnormalIOReactor}}, it will hang after the word "Done" > appears, which occurs after the .close() method is called on the httpclient. > When running jstack from the command line, I see that various "I/O > Dispatcher" threads are still running for the client even though I had > expected them to have been closed: > {noformat} > "I/O dispatcher 8" #19 prio=5 os_prio=31 tid=0x00007f978e800000 nid=0x6503 > runnable [0x0000000120cf6000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) > at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x000000076ba79bf8> (a sun.nio.ch.Util$2) > - locked <0x000000076ba79be8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000000076ba79ac8> (a sun.nio.ch.KQueueSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:257) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) > at > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I had been thinking that in this situation it would be better for the > httpclient to not be implicitly stopped - so it could be used for subsequent > requests if/when the number of open file descriptors has dropped. Also, it > would be good for either the httpclient.execute() call to throw an exception > right away or at least for calls to .get() on the future returned from the > .execute() call to throw an exception and for the .failed() method on > {{FutureCallback<HttpResponse>}} object to be called. > Note that I ran this application on my MacBook Pro, running OS-X Yosemite. > We've seen this problem on other OSes as well, though, including CentOS 7. > I reproduced this problem with a few different versions of HttpAsyncClient - > 4.0.2, 4.1.1, custom build from trunk, custom build from the 4.1.x branch. > This isn't a situation that we run into often with our applications - mostly > in cases where users haven't tuned their application settings properly to > account for the number of file descriptors that a Java process would need to > use. It would be nice if the HttpAsyncClient handling could gracefully fail > a request when open file descriptors have temporarily been exhausted under > load spikes but leave the client in a usable state for future requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org For additional commands, e-mail: dev-h...@hc.apache.org