Peter Lebedev created SOLR-16084:
------------------------------------
Summary: When Solr server runs with HTTPS java clients can't
connect but curl can.
Key: SOLR-16084
URL: https://issues.apache.org/jira/browse/SOLR-16084
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: clients - java
Environment: java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment Temurin-17.0.2+8 (build 17.0.2+8)
OpenJDK 64-Bit Server VM Temurin-17.0.2+8 (build 17.0.2+8, mixed mode, sharing)
The application is using spring boot 2.5.6, solr-solrj 8.11.1, and apache
httpclient 4.5.13. It is also forcing TLS 1.2.
Reporter: Peter Lebedev
When we have Solr server running on HTTPS, `Http2SolrClient` is not able to
communicate with it, while curl works, and apache HTTP client,
[org|eclipse-javadoc:%E2%98%82=job-search-service/%5C/Users%5C/plebedev%5C/.m2%5C/repository%5C/org%5C/apache%5C/httpcomponents%5C/httpclient%5C/4.5.13%5C/httpclient-4.5.13.jar=/maven.pomderived=/true=/=/maven.pomderived=/true=/=/maven.groupId=/org.apache.httpcomponents=/=/maven.artifactId=/httpclient=/=/maven.version=/4.5.13=/=/maven.scope=/compile=/%3Corg].[apache|eclipse-javadoc:%E2%98%82=job-search-service/%5C/Users%5C/plebedev%5C/.m2%5C/repository%5C/org%5C/apache%5C/httpcomponents%5C/httpclient%5C/4.5.13%5C/httpclient-4.5.13.jar=/maven.pomderived=/true=/=/maven.pomderived=/true=/=/maven.groupId=/org.apache.httpcomponents=/=/maven.artifactId=/httpclient=/=/maven.version=/4.5.13=/=/maven.scope=/compile=/%3Corg.apache].[http|eclipse-javadoc:%E2%98%82=job-search-service/%5C/Users%5C/plebedev%5C/.m2%5C/repository%5C/org%5C/apache%5C/httpcomponents%5C/httpclient%5C/4.5.13%5C/httpclient-4.5.13.jar=/maven.pomderived=/true=/=/maven.pomderived=/true=/=/maven.groupId=/org.apache.httpcomponents=/=/maven.artifactId=/httpclient=/=/maven.version=/4.5.13=/=/maven.scope=/compile=/%3Corg.apache.http].[impl|eclipse-javadoc:%E2%98%82=job-search-service/%5C/Users%5C/plebedev%5C/.m2%5C/repository%5C/org%5C/apache%5C/httpcomponents%5C/httpclient%5C/4.5.13%5C/httpclient-4.5.13.jar=/maven.pomderived=/true=/=/maven.pomderived=/true=/=/maven.groupId=/org.apache.httpcomponents=/=/maven.artifactId=/httpclient=/=/maven.version=/4.5.13=/=/maven.scope=/compile=/%3Corg.apache.http.impl].[client|eclipse-javadoc:%E2%98%82=job-search-service/%5C/Users%5C/plebedev%5C/.m2%5C/repository%5C/org%5C/apache%5C/httpcomponents%5C/httpclient%5C/4.5.13%5C/httpclient-4.5.13.jar=/maven.pomderived=/true=/=/maven.pomderived=/true=/=/maven.groupId=/org.apache.httpcomponents=/=/maven.artifactId=/httpclient=/=/maven.version=/4.5.13=/=/maven.scope=/compile=/%3Corg.apache.http.impl.client].CloseableHttpClient
if called from the same Java application.
Here is the application level exception:
{code:java}
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException
occured when talking to server at:
https://jobcase-solr-job-prod-replica.sensitive-marketplace-staging.ojop.io/solr
78 at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:437)
~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
79 at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
80 at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
89 ... 61 more
90Caused by: java.io.IOException: Broken pipe
91 at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1135)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
92 at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
93 at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
94 at
org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
95 at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:295)
~[http2-common-9.4.44.v20210927.jar!/:9.4.44.v20210927]
96 at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
~[jetty-util-9.4.44.v20210927.jar!/:9.4.44.v20210927]
97 at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
~[jetty-util-9.4.44.v20210927.jar!/:9.4.44.v20210927]
98 at org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:729)
~[http2-common-9.4.44.v20210927.jar!/:9.4.44.v20210927]
99 at org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:701)
~[http2-common-9.4.44.v20210927.jar!/:9.4.44.v20210927]
100 at
org.eclipse.jetty.http2.client.HTTP2ClientConnectionFactory$HTTP2ClientConnection.onOpen(HTTP2ClientConnectionFactory.java:116)
~[http2-client-9.4.44.v20210927.jar!/:9.4.44.v20210927]
101 at
org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:444)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
102 at
org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:114)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
103 at
org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:84)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
104 at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
105 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
106 at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
107 at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
108 at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
109 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
110 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
111 at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
112 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
~[?:?]
113 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
~[?:?] {code}
When I enable ssl debug, I see this in the logs:
{code:java}
javax.net.ssl|ERROR|7B|h2sc-1-thread-5|2022-03-04 15:15:22.809
EST|TransportContext.java:363|Fatal (INTERNAL_ERROR): closing inbound before
receiving peer's close_notify (
546"throwable" : {
547 javax.net.ssl.SSLException: closing inbound before receiving peer's
close_notify
548 at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
549 at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
550 at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358)
551 at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
552 at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:305)
553 at
java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:796)
554 at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.closeInbound(SslConnection.java:982)
555 at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:766)
556 at
org.eclipse.jetty.io.NegotiatingClientConnection.fill(NegotiatingClientConnection.java:99)
557 at
org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:81)
558 at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
559 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
560 at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
561 at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
562 at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
563 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
564 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
565 at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
566 at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
567 at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
568 at java.base/java.lang.Thread.run(Thread.java:833)} {code}
This seems to be related to
[https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8253368]
How can I make it working with https Solr server?
It is possible that the issue is with jetty HTTP client, not with Solr client
directly, so it could be a misconfiguration. However, there is no way to
provide externally configured jetty http client, nor it supports the same level
of configuration as apache http client. For example, when I build apache
HttpClient I set these on its builder:
{code:java}
builder.setSSLHostnameVerifier(hostnameVerifier);
builder.setSSLContext(sslContext);
builder.setSSLSocketFactory(sslConnSocketFactory); {code}
and https connections work to the same Solr server.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
