Hello, I am currently setting up a Solr cluster using version 9.7. Everything is working quite well, except for one issue. I’ve created a simple example to reproduce the problem. I have a docker-compose.yml file with the following content:
docker-compose.yml
version: '3'
services:
solr:
image: solr:9.7
ports:
- "8983:8983"
- "5005:5005"
environment:
ZK_HOST: "zoo:2181"
depends_on: [zoo]
solr2:
image: solr:9.7
ports:
- "8984:8983"
environment:
ZK_HOST: "zoo:2181"
depends_on: [zoo]
zoo:
image: zookeeper:3.9
ports:
- "2181:2181"
environment:
ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok"
I start the server, and the web interface is accessible at
http://localhost:8983/solr/.
Then I create two collections:
curl
"http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
curl
"http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test2&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
This works, and I can send the following requests:
curl "http://localhost:8983/solr/Test/config/overlay";
{
"responseHeader":{
"status":0,
"QTime":28
},
"overlay":{
"znodeVersion":-1
}
}%
curl "http://localhost:8983/solr/Test2/config/overlay";
{
"responseHeader":{
"status":0,
"QTime":25
},
"overlay":{
"znodeVersion":-1
}
}%
This works as well. Collection Test is on one node, and Test2 is on the other.
One of the requests gets forwarded from 172.20.0.4:8983 to 172.20.0.3:8983
However, when I use HTTP/2, the forwarded request hangs for 30 seconds and then
fails:
curl --http2 "http://localhost:8983/solr/Test2/config/overlay";
curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
curl --http2 "http://localhost:8983/solr/Test/config/overlay";
{
"responseHeader":{
"status":0,
"QTime":2
},
"overlay":{
"znodeVersion":-1
}
In the logs, I find the following exception:
2024-10-17 10:26:05.536 ERROR (qtp1677458155-20-null-79) [c: s: r: x:
t:null-79] o.a.s.s.HttpSolrCall 500 Exception =>
org.apache.solr.common.SolrException: Error trying to proxy request for url:
http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817)
solr-1 | org.apache.solr.common.SolrException: Error trying to proxy request
for url: http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817) ~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.sendRemoteQuery(HttpSolrCall.java:504)
~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) ~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:251)
~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:208)
~[?:?]
solr-1 | at
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:243)
~[?:?]
solr-1 | at
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:213)
~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
~[?:?]
solr-1 | at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
~[jetty-security-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
~[jetty-rewrite-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at org.eclipse.jetty.server.Server.handle(Server.java:563)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208)
~[http2-common-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.HTTP2Connection.offerTask(HTTP2Connection.java:193)
~[http2-common-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerConnection.onNewStream(HTTP2ServerConnection.java:144)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onNewStream(HTTP2ServerConnectionFactory.java:97)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.HTTP2Session.notifyNewStream(HTTP2Session.java:1136)
~[http2-common-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerSession.onHeaders(HTTP2ServerSession.java:120)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerSession.onFrame(HTTP2ServerSession.java:188)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerConnection.onOpen(HTTP2ServerConnection.java:113)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:451)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.upgrade(HttpConnection.java:424)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:455)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannelOverHttp.upgrade(HttpChannelOverHttp.java:677)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannelOverHttp.headerComplete(HttpChannelOverHttp.java:416)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http.HttpParser.parseFields(HttpParser.java:1256)
~[jetty-http-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1542)
~[jetty-http-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:403)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:275)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at java.base/java.lang.Thread.run(Unknown Source) [?:?]
solr-1 | Caused by: org.apache.http.client.ClientProtocolException
solr-1 | at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
solr-1 | ... 70 more
solr-1 | Caused by: org.apache.http.ProtocolException: The server failed to
respond with a valid HTTP response
solr-1 | at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:149)
~[?:?]
solr-1 | at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[?:?]
solr-1 | at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[?:?]
solr-1 | at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[?:?]
solr-1 | at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
~[?:?]
solr-1 | at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[?:?]
solr-1 | at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[?:?]
solr-1 | at
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:144)
~[?:?]
solr-1 | at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[?:?]
solr-1 | at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[?:?]
solr-1 | at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[?:?]
solr-1 | at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[?:?]
solr-1 | at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
solr-1 | ... 70 more
Is this a bug, or is HTTP/2 simply not supported yet for requests forwarded to
another node?
I found this in the source code, and it seems that HTTP/2 support is still
missing there:
HttpSolrCall: 729
// TODO using Http2Client
private void remoteQuery(String coreUrl, HttpServletResponse resp) throws
IOException {
Best regards,
Sebastian Hofmann
smime.p7s
Description: S/MIME cryptographic signature
