Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
Shawn, You should be able to add --module=http-forwarded to the command line to enable the ForwardedRequestCustomizer module. You should be able to add jetty.httpConfig.relativeRedirectAllowed=true to the command line to configure relative redirection. However, typically jetty start.jar likes to have persistent configuration for modules so that you can configure them in files. If you manually did: java -server -Djetty.home=/opt/solr/server -jar start.jar --add-module=server,http,gzip,http-forwarded then that should create a start.d directory with server.ini (and http.ini and http-forwarded.ini etc.) which you can edit to configure the server. cheers On Wed, 25 May 2022 at 12:03, Shawn Heisey wrote: > On 5/24/2022 4:34 PM, Greg Wilkins wrote: > > To turn on relative redirects, you need to set the property > > `jetty.httpConfig.relativeRedirectAllowed=true`, which can be done on > > the command line or better yet in the server.ini file. > > There is no server.ini file in a Solr install. How would I configure > that? We certainly could define a sysprop on the commandline, we > already do it for other things, but I would prefer a way to do it > through configuration rather than commandline parameters. The full > command that the script uses to start Solr is already quite long. This > is from a stock install of version 8.11.1: > > java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem > -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages > -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent > -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M > > -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes= > -Dsolr.log.dir=/var/solr/logs -Djetty.port=8981 -DSTOP.PORT=7981 > -DSTOP.KEY=solrrocks -Dhost=127.0.0.1 -Duser.timezone=UTC > -XX:-OmitStackTraceInFastThrow > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8981 /var/solr/logs > -Djetty.home=/opt/solr/server -Dsolr.solr.home=/var/solr/data > -Dsolr.data.home= -Dsolr.install.dir=/opt/solr > -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf > -Dlog4j.configurationFile=/var/solr/log4j2.xml -Xss256k > -Dsolr.log.muteconsole -jar start.jar --module=http --module=gzip > > Would you expect configuring ForwardedRequestCustomizer in our provided > jetty.xml (or one of the other xml configs when using SSL) to cause > problems? If I understand what it does correctly, I would not expect it > to cause problems for users who do not have a proxy, but I haven't > looked at the code and I am hoping that someone here knows enough about > it that they can answer that question. > > I am inclined right now to uncomment the ForwardedRequestCustomizer > section in our xml configs for our next release, if someone can tell me > that it would be a safe thing to do. I've done some minimal testing and > it did not appear to cause any issues. But it certainly was not an > extensive test. Maybe I can find a co-conspirator with a large > development or testing SolrCloud cluster that would be willing to try it > out. > > Thanks, > Shawn > > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > -- Greg Wilkins CTO http://webtide.com ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
On 5/24/2022 4:34 PM, Greg Wilkins wrote: To turn on relative redirects, you need to set the property `jetty.httpConfig.relativeRedirectAllowed=true`, which can be done on the command line or better yet in the server.ini file. There is no server.ini file in a Solr install. How would I configure that? We certainly could define a sysprop on the commandline, we already do it for other things, but I would prefer a way to do it through configuration rather than commandline parameters. The full command that the script uses to start Solr is already quite long. This is from a stock install of version 8.11.1: java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes= -Dsolr.log.dir=/var/solr/logs -Djetty.port=8981 -DSTOP.PORT=7981 -DSTOP.KEY=solrrocks -Dhost=127.0.0.1 -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8981 /var/solr/logs -Djetty.home=/opt/solr/server -Dsolr.solr.home=/var/solr/data -Dsolr.data.home= -Dsolr.install.dir=/opt/solr -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf -Dlog4j.configurationFile=/var/solr/log4j2.xml -Xss256k -Dsolr.log.muteconsole -jar start.jar --module=http --module=gzip Would you expect configuring ForwardedRequestCustomizer in our provided jetty.xml (or one of the other xml configs when using SSL) to cause problems? If I understand what it does correctly, I would not expect it to cause problems for users who do not have a proxy, but I haven't looked at the code and I am hoping that someone here knows enough about it that they can answer that question. I am inclined right now to uncomment the ForwardedRequestCustomizer section in our xml configs for our next release, if someone can tell me that it would be a safe thing to do. I've done some minimal testing and it did not appear to cause any issues. But it certainly was not an extensive test. Maybe I can find a co-conspirator with a large development or testing SolrCloud cluster that would be willing to try it out. Thanks, Shawn ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
Shawn, I think you may have a solution now, but I'll just answer any outstanding questions just in case. To turn on relative redirects, you need to set the property `jetty.httpConfig.relativeRedirectAllowed=true`, which can be done on the command line or better yet in the server.ini file. The example you sent without a proxy is exactly correct behaviour as the Host header sent is used for the non relative redirection. As you have discovered, either the Host header must be the original from the client OR the ForwardedRequestCustomizer must be used to pass over the same information. So I think it is worthwhile to correctly configure your proxy anyway, as that reduces the possibility of anything else leaking your internal IP addresses. and then configure relative redirects anyway so you have defence in depth. cheers On Wed, 25 May 2022 at 02:31, Shawn Heisey wrote: > On 5/24/22 02:03, Greg Wilkins wrote: > > To say more, we'd need to see the headers of the request arriving at > > the proxy and then arriving at jetty but my money is on their > > proxy being configured to rewrite the host header. > > I was able to duplicate the issue, and I did not have haproxy configured > to do any kind of rewriting. > > This is what I get with a verbose curl: > > https://paste.elyograg.org/view/e95b70a0 > > This is what haproxy logged for that request, showing a 302 response: > > May 24 10:01:41 - haproxy[299524] 192.168.217.199:59602 > [24/May/2022:10:01:41.656] solr~ be-solr/g8981 0/0/0/1/1 302 105 - - > --NI 1/1/0/0/0 0/0 "GET https://solr.elyograg.org:8983/ HTTP/2.0" > > Where would I do HttpConfiguration.setRelativeRedirectAllowed(true)? > That looks like Java code, and Jetty is not embedded. This is the > jetty.xml in Solr 8.x: > > > https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=blob_plain;f=solr/server/etc/jetty.xml;h=e2f4ab095984aac27185a9879964862f9ba35d4d;hb=refs/heads/branch_8_11 > > I'm still digesting Uwe's reply. To answer his question: The proxy is > https, Solr is http. > > The relevant parts of my haproxy config can be found at the link below. > The commented lines in the frontend are how I worked around the issue in > haproxy -- handling root path redirect in haproxy rather than letting it > through to Solr: > > https://paste.elyograg.org/view/b3b413c3 > > Thanks, > Shawn > > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > -- Greg Wilkins CTO http://webtide.com ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
On 5/24/22 12:30, Joakim Erdfelt wrote: If you are going to use the ForwardedRequestCustomizer do yourself a favor and _only use the RFC 7239 mode_. Do not use the X-Forwarded-* headers, they are a giant mess of conflicting behaviors and rules with no standardization behind them. Big players interpret the headers differently (Google, vs AWS, vs Azure, vs haproxy, etc) Only use the `Forwarded` header, per the https://tools.ietf.org/html/rfc7239 spec. You'll eliminate so many support issues just by doing this. Very cool! I had to look that up. Nice that there is a true standard for proxy headers. I turned off the X-Forwarded-For header for that frontend and changed the line in the haproxy config adding X-Forwarded-Proto to this and it still works: http-request add-header Forwarded "for=\"%[src]\"; proto=https" The info I found about using that RFC with haproxy had the proto setting referencing a variable, but that didn't work. Had to hardcode it to https. Which is OK, as the frontend only does https. Thanks, Shawn ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
If you are going to use the ForwardedRequestCustomizer do yourself a favor and *only use the RFC 7239 mode*. Do not use the X-Forwarded-* headers, they are a giant mess of conflicting behaviors and rules with no standardization behind them. Big players interpret the headers differently (Google, vs AWS, vs Azure, vs haproxy, etc) Only use the `Forwarded` header, per the https://tools.ietf.org/html/rfc7239 spec. You'll eliminate so many support issues just by doing this. Joakim Erdfelt / joa...@webtide.com On Tue, May 24, 2022 at 12:09 PM Shawn Heisey wrote: > On 5/24/22 10:31, Shawn Heisey wrote: > > Where would I do HttpConfiguration.setRelativeRedirectAllowed(true)? > > That looks like Java code, and Jetty is not embedded. This is the > > jetty.xml in Solr 8.x: > > > > > https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=blob_plain;f=solr/server/etc/jetty.xml;h=e2f4ab095984aac27185a9879964862f9ba35d4d;hb=refs/heads/branch_8_11 > > If we incorporated the ForwardedRequestCustomizer that Uwe mentioned in > our jetty config, would that work for those using a proxy as well as > those NOT using a proxy? I see that this config is already in our > jetty.xml, commented. > > What I am hoping for is coming up with an xml config that allows this to > work properly for all users whether they incorporate a proxy or not. > Most proxies send X-Forwarded-For, and configuring X-Forwarded-Proto is > typically very easy. > > I uncommented the call to ForwardedRequestCustomizer and configured > haproxy to send the X-Forwarded-Proto header. This fixed the redirect, > and it looks like a request sent directly to Solr still works properly > too. Can anyone imagine any unintended side effects from uncommenting > ForwardedRequestCustomizer in our source repo's XML configs for jetty? > > Is there a more generic fix that would cause jetty to NOT send an > absolute URL path in its redirect? (what I was initially aiming for) > > Thanks, > Shawn > > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
On 5/24/22 10:31, Shawn Heisey wrote: Where would I do HttpConfiguration.setRelativeRedirectAllowed(true)? That looks like Java code, and Jetty is not embedded. This is the jetty.xml in Solr 8.x: https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=blob_plain;f=solr/server/etc/jetty.xml;h=e2f4ab095984aac27185a9879964862f9ba35d4d;hb=refs/heads/branch_8_11 If we incorporated the ForwardedRequestCustomizer that Uwe mentioned in our jetty config, would that work for those using a proxy as well as those NOT using a proxy? I see that this config is already in our jetty.xml, commented. What I am hoping for is coming up with an xml config that allows this to work properly for all users whether they incorporate a proxy or not. Most proxies send X-Forwarded-For, and configuring X-Forwarded-Proto is typically very easy. I uncommented the call to ForwardedRequestCustomizer and configured haproxy to send the X-Forwarded-Proto header. This fixed the redirect, and it looks like a request sent directly to Solr still works properly too. Can anyone imagine any unintended side effects from uncommenting ForwardedRequestCustomizer in our source repo's XML configs for jetty? Is there a more generic fix that would cause jetty to NOT send an absolute URL path in its redirect? (what I was initially aiming for) Thanks, Shawn ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
On 5/24/22 02:03, Greg Wilkins wrote: To say more, we'd need to see the headers of the request arriving at the proxy and then arriving at jetty but my money is on their proxy being configured to rewrite the host header. I was able to duplicate the issue, and I did not have haproxy configured to do any kind of rewriting. This is what I get with a verbose curl: https://paste.elyograg.org/view/e95b70a0 This is what haproxy logged for that request, showing a 302 response: May 24 10:01:41 - haproxy[299524] 192.168.217.199:59602 [24/May/2022:10:01:41.656] solr~ be-solr/g8981 0/0/0/1/1 302 105 - - --NI 1/1/0/0/0 0/0 "GET https://solr.elyograg.org:8983/ HTTP/2.0" Where would I do HttpConfiguration.setRelativeRedirectAllowed(true)? That looks like Java code, and Jetty is not embedded. This is the jetty.xml in Solr 8.x: https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=blob_plain;f=solr/server/etc/jetty.xml;h=e2f4ab095984aac27185a9879964862f9ba35d4d;hb=refs/heads/branch_8_11 I'm still digesting Uwe's reply. To answer his question: The proxy is https, Solr is http. The relevant parts of my haproxy config can be found at the link below. The commented lines in the frontend are how I worked around the issue in haproxy -- handling root path redirect in haproxy rather than letting it through to Solr: https://paste.elyograg.org/view/b3b413c3 Thanks, Shawn ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
Hi Shawn, as Greg Wilkins already said, there are often problems with proxies not transferring the correct information to backend. What I was not able to figure out from the Solr issue: Is only the proxy HTTPS and Solr is HTTP, or are both HTTP? If Solr works with HTTP and the TLS termination is only done by the proxy, then it is definitely a configuration problem of both Solr and Proxy: - You need to enable the reverse proxy customizer in Solr: https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/ForwardedRequestCustomizer.html - This one gets the special proxy "X-Forwarded-..." headers and sets the internal request parameters inside Jetty correct - The proxy has to be configured to send above headers, especially the "X-Forwarded-Proto" needs to be set. Apache does not do this by default, nginx does (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto). Uwe Am 24.05.2022 um 06:52 schrieb Shawn Heisey: I'm with the Solr project. Solr runs in Jetty. It is not embedded, the startup script runs start.jar. I'm trying to help one of our users with a problem. They have a proxy server handling TLS, Solr listens without encryption. Here's the bug report on Solr: https://issues.apache.org/jira/browse/SOLR-16200/ If they access the admin UI via the proxy server using https on the /solr URL path, where the webapp lives, everything's fine. All links stay https even though Solr is listening without encryption. But if they access the root URL with https, Jetty is redirecting them to a URL that is hardcoded as http://server:port/solr/ instead of preserving the https:// that was used to access it, which doesn't work, because the proxy is configured with TLS. I have reproduced the problem. I found a way to work around the issue with my proxy server, but it would be really nice if that was not required. Is there any way in the jetty config to have it preserve the URL scheme when it redirects? If it were to redirect to a relative path of /solr instead of an absolute URL that contains http:// that would fix it. What information do I need to provide to help troubleshoot and find a fix? Thanks, Shawn ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users -- Uwe Schindler uschind...@apache.org ASF Member, Member of PMC and Committer of Apache Lucene and Apache Solr Bremen, Germany https://lucene.apache.org/ https://solr.apache.org/ ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Solr admin UI redirecting root path requests to http instead of https
Shawn,
Have you tried configuring
HttpConfiguration.setRelativeRedirectAllowed(true) ?
This should stop Jetty trying to make a relative redirect into an absolute
one and perhaps using the wrong authority.
However, even without this, the server should get the authority correct if
the proxy is being well behaved. Some proxies rewrite the Host header,
which entirely defeats the purpose of the Host header (to tell the server
what authority the original client used), so you can look at the
configuration of your proxy.
Alternatively, it could be that there is no authority at all? ie no host
header and a HTTP/1.0 request?
To say more, we'd need to see the headers of the request arriving at the
proxy and then arriving at jetty but my money is on their proxy being
configured to rewrite the host header.
cheers
PS. If they really can't fix their proxy, there are ways that we can force
jetty to use a specific authority even when the client doesn't give one
or just use relative redirects.
On Tue, 24 May 2022 at 14:52, Shawn Heisey wrote:
> I'm with the Solr project. Solr runs in Jetty. It is not embedded, the
> startup script runs start.jar.
>
> I'm trying to help one of our users with a problem. They have a proxy
> server handling TLS, Solr listens without encryption.
>
> Here's the bug report on Solr:
>
> https://issues.apache.org/jira/browse/SOLR-16200/
>
> If they access the admin UI via the proxy server using https on the
> /solr URL path, where the webapp lives, everything's fine. All links
> stay https even though Solr is listening without encryption. But if
> they access the root URL with https, Jetty is redirecting them to a URL
> that is hardcoded as http://server:port/solr/ instead of preserving the
> https:// that was used to access it, which doesn't work, because the
> proxy is configured with TLS. I have reproduced the problem. I found a
> way to work around the issue with my proxy server, but it would be
> really nice if that was not required.
>
> Is there any way in the jetty config to have it preserve the URL scheme
> when it redirects? If it were to redirect to a relative path of /solr
> instead of an absolute URL that contains http:// that would fix it.
>
> What information do I need to provide to help troubleshoot and find a fix?
>
> Thanks,
> Shawn
>
> ___
> jetty-users mailing list
> jetty-users@eclipse.org
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jetty-users
>
--
Greg Wilkins CTO http://webtide.com
//
//
// Copyright (c) 1995-2022 Mort Bay Consulting Pty Ltd and others.
//
// This program and the accompanying materials are made available under the
// terms of the Eclipse Public License v. 2.0 which is available at
// https://www.eclipse.org/legal/epl-2.0, or the Apache License, Version 2.0
// which is available at https://www.apache.org/licenses/LICENSE-2.0.
//
// SPDX-License-Identifier: EPL-2.0 OR Apache-2.0
//
//
package org.eclipse.jetty.server;
import java.io.IOException;
import java.net.SocketAddress;
import java.util.List;
import java.util.Set;
import java.util.concurrent.CopyOnWriteArrayList;
import org.eclipse.jetty.http.CookieCompliance;
import org.eclipse.jetty.http.HttpCompliance;
import org.eclipse.jetty.http.HttpMethod;
import org.eclipse.jetty.http.HttpScheme;
import org.eclipse.jetty.http.UriCompliance;
import org.eclipse.jetty.util.HostPort;
import org.eclipse.jetty.util.Index;
import org.eclipse.jetty.util.Jetty;
import org.eclipse.jetty.util.annotation.ManagedAttribute;
import org.eclipse.jetty.util.annotation.ManagedObject;
import org.eclipse.jetty.util.component.Dumpable;
import org.eclipse.jetty.util.component.DumpableCollection;
/**
* HTTP Configuration.
* This class is a holder of HTTP configuration for use by the
* {@link HttpChannel} class. Typically an HTTPConfiguration instance
* is instantiated and passed to a {@link ConnectionFactory} that can
* create HTTP channels (e.g. HTTP, AJP or FCGI).
* The configuration held by this class is not for the wire protocol,
* but for the interpretation and handling of HTTP requests that could
* be transported by a variety of protocols.
*
*/
@ManagedObject("HTTP Configuration")
public class HttpConfiguration implements Dumpable
{
public static final String SERVER_VERSION = "Jetty(" + Jetty.VERSION + ")";
private final List _customizers = new CopyOnWriteArrayList<>();
private final Index.Mutable _formEncodedMethods = new Index.Builder()
.caseSensitive(false)
.mutable()
.build();
private int _outputBufferSize = 32 * 1024;
private int _outputAggregationSize = _outputBufferSize / 4;
private int _requestHeaderSize = 8 * 1024;
private int _responseHeaderSize = 8 * 1024;
private int _headerCacheSize = 1024;
