These links are not providing solutions but may be provide some ideas for the investigation. I suggest to try the -Djavax.net.debug=all JVM parameter for your client application.
Good luke. Dominique Le lun. 17 août 2020 à 19:11, Odysci <ody...@gmail.com> a écrit : > Dominique, > thanks, but I'm not sure the links you sent point to an actual solution. > The Nginx logs, sometimes give a 499 return code which is: > (499 Client Closed Request > Used when the client has closed the request before the server could send a > response. > > but the timestamps of these log msgs do not coincide with the IOException, > so I'm not sure they are related. > Reinaldo > > On Mon, Aug 17, 2020 at 12:59 PM Dominique Bejean < > dominique.bej...@eolya.fr> wrote: > >> Hi, >> >> It looks like this issues >> https://github.com/eclipse/jetty.project/issues/4883 >> https://github.com/eclipse/jetty.project/issues/2571 >> >> The Nginx server closed the connection. Any info in nginx log ? >> >> Dominique >> >> Le lun. 17 août 2020 à 17:33, Odysci <ody...@gmail.com> a écrit : >> >>> Hi, >>> thanks for the reply. >>> We're using solr 8.3.1, ZK 3.5.6 >>> The stacktrace is below. >>> The address on the first line " >>> http://192.168.15.10:888/solr/mycollection"; is the "server" address in >>> my nginx configuration, which points to 2 upstream solr nodes. There were >>> no other solr or ZK messages in the logs. >>> >>> StackTrace: >>> (Msg = IOException occured when talking to server at: >>> http://192.168.15.10:888/solr/mycollection) >>> org.apache.solr.client.solrj.SolrServerException: IOException occured >>> when talking to server at: http://192.168.15.10:888/solr/mycollection >>> at >>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418) >>> at >>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754) >>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211) >>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035) >>> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051) >>> ... calls from our code >>> ... calls from our code >>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>> at >>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> at java.base/java.lang.Thread.run(Thread.java:834) >>> Caused by: java.nio.channels.AsynchronousCloseException >>> at >>> org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144) >>> at >>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170) >>> at >>> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232) >>> at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206) >>> at >>> org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153) >>> at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438) >>> at >>> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392) >>> at >>> org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187) >>> at >>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169) >>> at >>> org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108) >>> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194) >>> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123) >>> at >>> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) >>> at >>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170) >>> at >>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125) >>> at >>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348) >>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) >>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) >>> at >>> org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) >>> at >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) >>> at >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) >>> ... 3 more >>> >>> ----------------------------- >>> >>> I did consider using the solrJ cloud or lb clients, but nginx gives me >>> more flexibility in controlling how the load balancing is done. I'm still >>> running experiments to see which one works best for me. >>> In the meantime, if you have any clues for why I'm getting this >>> IOException, I'd appreciate. >>> Thanks >>> >>> Reinaldo >>> >>> >>> >>> On Mon, Aug 17, 2020 at 10:59 AM Dominique Bejean < >>> dominique.bej...@eolya.fr> wrote: >>> >>>> Hi, >>>> >>>> >>>> >>>> >>>> >>>> Can you provide more information ? >>>> >>>> >>>> - Solr and ZK version >>>> >>>> >>>> - full error stacktrace generated by SolrJ >>>> >>>> >>>> - any concomitant and relevant information in solr nodes logs or zk logs >>>> >>>> >>>> >>>> >>>> >>>> Just to know, why not use a load balanced LBHttp... Solr Client ? >>>> >>>> >>>> >>>> >>>> >>>> Regards. >>>> >>>> >>>> >>>> >>>> >>>> Dominique >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Le lun. 17 août 2020 à 00:41, Odysci <ody...@gmail.com> a écrit : >>>> >>>> >>>> >>>> >>>> >>>> > Hi, >>>> >>>> >>>> > >>>> >>>> >>>> > We have a solrcloud setup with 2 solr nodes and 3 ZK instances. Until >>>> >>>> >>>> > recently I had my application server always call one of the solr >>>> nodes (via >>>> >>>> >>>> > solrJ), and it worked just fine. >>>> >>>> >>>> > >>>> >>>> >>>> > In order to improve reliability I put an Nginx reverse-proxy load >>>> balancer >>>> >>>> >>>> > between my application server and the solr nodes. The response time >>>> >>>> >>>> > remained almost the same but we started getting the msg: >>>> >>>> >>>> > >>>> >>>> >>>> > IOException occured when talking to server http://myserver.... >>>> >>>> >>>> > >>>> >>>> >>>> > every minute or so (very randomly but consistently). Since our code >>>> will >>>> >>>> >>>> > just try again after a few milliseconds, the overall system continues >>>> to >>>> >>>> >>>> > work fine, despite the delay. I tried increasing all nginx related >>>> >>>> >>>> > timeout's to no avail. >>>> >>>> >>>> > I've searched for this msg a lot and most replies seem to be related >>>> to >>>> >>>> >>>> > ssl. >>>> >>>> >>>> > We are using http2solrclient but no ssl to solr. >>>> >>>> >>>> > Can anyone shed any light on this? >>>> >>>> >>>> > >>>> >>>> >>>> > Thanks! >>>> >>>> >>>> > Reinaldo >>>> >>>> >>>> > >>>> >>>> >>>> >>> >>>
