Hi Sebastian,
This does sound like a "bug" - would you be willing to file a JIRA
ticket with this information please? ("Bonus points" if you can tweak
your reproduction steps to use "bin/solr start -e cloud" instead of
docker-compose, as that's probably more approachable for some folks!
But no worries if not.)
Thanks for raising this!
Best,
Jason
On Thu, Oct 17, 2024 at 7:45 AM Sebastian Hofmann
<[email protected]> wrote:
>
> Hello,
>
> I am currently setting up a Solr cluster using version 9.7. Everything is
> working quite well, except for one issue. I’ve created a simple example to
> reproduce the problem. I have a docker-compose.yml file with the following
> content:
>
> docker-compose.yml
> version: '3'
> services:
> solr:
> image: solr:9.7
> ports:
> - "8983:8983"
> - "5005:5005"
> environment:
> ZK_HOST: "zoo:2181"
> depends_on: [zoo]
> solr2:
> image: solr:9.7
> ports:
> - "8984:8983"
> environment:
> ZK_HOST: "zoo:2181"
> depends_on: [zoo]
> zoo:
> image: zookeeper:3.9
> ports:
> - "2181:2181"
> environment:
> ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok"
>
> I start the server, and the web interface is accessible at
> http://localhost:8983/solr/.
>
> Then I create two collections:
>
> curl
> "http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
>
> curl
> "http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test2&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
>
> This works, and I can send the following requests:
>
> curl "http://localhost:8983/solr/Test/config/overlay";
>
> {
> "responseHeader":{
> "status":0,
> "QTime":28
> },
> "overlay":{
> "znodeVersion":-1
> }
> }%
>
>
> curl "http://localhost:8983/solr/Test2/config/overlay";
>
> {
> "responseHeader":{
> "status":0,
> "QTime":25
> },
> "overlay":{
> "znodeVersion":-1
> }
> }%
>
> This works as well. Collection Test is on one node, and Test2 is on the
> other. One of the requests gets forwarded from 172.20.0.4:8983 to
> 172.20.0.3:8983
>
> However, when I use HTTP/2, the forwarded request hangs for 30 seconds and
> then fails:
>
> curl --http2 "http://localhost:8983/solr/Test2/config/overlay";
> curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
>
> curl --http2 "http://localhost:8983/solr/Test/config/overlay";
> {
> "responseHeader":{
> "status":0,
> "QTime":2
> },
> "overlay":{
> "znodeVersion":-1
> }
>
> In the logs, I find the following exception:
> 2024-10-17 10:26:05.536 ERROR (qtp1677458155-20-null-79) [c: s: r: x:
> t:null-79] o.a.s.s.HttpSolrCall 500 Exception =>
> org.apache.solr.common.SolrException: Error trying to proxy request for url:
> http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817)
> solr-1 | org.apache.solr.common.SolrException: Error trying to proxy
> request for url: http://172.20.0.3:8983/solr/Test2/config/overlay with
> _forwardCount: 1
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817) ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.sendRemoteQuery(HttpSolrCall.java:504)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:251)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:208)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:243)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:213)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> ~[?:?]
> solr-1 | at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
> ~[jetty-security-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
> ~[jetty-rewrite-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.Server.handle(Server.java:563)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208)
> ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.HTTP2Connection.offerTask(HTTP2Connection.java:193)
> ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerConnection.onNewStream(HTTP2ServerConnection.java:144)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onNewStream(HTTP2ServerConnectionFactory.java:97)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.HTTP2Session.notifyNewStream(HTTP2Session.java:1136)
> ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerSession.onHeaders(HTTP2ServerSession.java:120)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerSession.onFrame(HTTP2ServerSession.java:188)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerConnection.onOpen(HTTP2ServerConnection.java:113)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:451)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.upgrade(HttpConnection.java:424)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:455)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannelOverHttp.upgrade(HttpChannelOverHttp.java:677)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannelOverHttp.headerComplete(HttpChannelOverHttp.java:416)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http.HttpParser.parseFields(HttpParser.java:1256)
> ~[jetty-http-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1542)
> ~[jetty-http-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:403)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:275)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at java.base/java.lang.Thread.run(Unknown Source) [?:?]
> solr-1 | Caused by: org.apache.http.client.ClientProtocolException
> solr-1 | at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
> solr-1 | ... 70 more
> solr-1 | Caused by: org.apache.http.ProtocolException: The server failed to
> respond with a valid HTTP response
> solr-1 | at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:149)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
> ~[?:?]
> solr-1 | at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
> ~[?:?]
> solr-1 | at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> ~[?:?]
> solr-1 | at
> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:144)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
> solr-1 | ... 70 more
>
>
> Is this a bug, or is HTTP/2 simply not supported yet for requests forwarded
> to another node?
>
> I found this in the source code, and it seems that HTTP/2 support is still
> missing there:
> HttpSolrCall: 729
>
> // TODO using Http2Client
> private void remoteQuery(String coreUrl, HttpServletResponse resp) throws
> IOException {
>
> Best regards,
> Sebastian Hofmann
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
