Re: Question on Apache Tomcat Patches
On 06/02/2020 20:28, Walker, Mike (GE Aviation, US) wrote: > HI All- > > New to apache and apache patching. I have a question. When you run an > apache patch (not an upgrade or install), should the patch create a new > apache folder path, or simply modify & update the existing one? > > For example, if I am updating apache tomcat 7 to 7_99, should I expect to see > a new folder C:\Program Files\Apache Software Foundation\Tomcat > 7.0_Tomcat7_99, or should I just see updates & modifications made to the > existing folder C:\Program Files\Apache Software Foundation\Tomcat 7.0 ? "Apache" normally means the "Apache HTTP Server Project" or "httpd". I'm guessing from your second paragraph you are referring to Apache Tomcat - in which case you are in the right place. The Apache Tomcat team does not distribute patches - we only distribute full versions. There are options for how to upgrade but none of them provided by the Tomcat community are automated. To help you (or point you in the right direction for help) we need to know: - what version of Apache Tomcat are you currently using? - where did you download this version from and was it a .zip, .exe or something else? - what version of Apache Tomcat do you want to move to? - where did you download this version from and is it a .zip, .exe or something else? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RewriteValve does not work on HTTPS
On 06/02/2020 23:00, Mark Thomas wrote:
> The issue appears to be that the following RewriteCond does not match
> when the request is served over HTTP/2 (with TLS) but it does if the
> request is served over HTTP/1.1 with TLS.
>
> RewriteCond %{HTTP_HOST} localhost
>
> (Note the RewriteCond quoted at the start of this thread is invalid)
>
> I'm wondering if this is related to the thread about
> HttpServletRequest.getRemoteAddr() returning null at the start of a
> request. It looks as if the request information may be being populated
> too late.
Nope. Unrelated.
The root cause here is that the RewriteCond is looking for an HTTP Host
header but HTTP/2 does not have any such header.
HTTP/2 has the pseudo-header ":authority" that replaces the Host header.
While you could argue that a strict interpretation of the mod_rewrite
docs (since the rewrite valve aims to emulate a sub-set of mod_rewrite
behaviour) means that "host" != ":authority" I think the reasonable
thing to do in this instance is to look at the ":authority" value if the
request is using HTTP/2. I'll work on a patch.
Mark
>
> Mark
>
>
>
> On 06/02/2020 20:42, Pierre Chiu wrote:
>> I just want to concur I have the same issue.
>> Removing that one line upgradeprotocol
>> and rewrite works again.
>>
>>
>>
>> On Thu, Feb 6, 2020 at 3:28 PM Hua Zhang wrote:
>>
>>> Nope, I also tested it with NIO instead of APR, same issues happen.
>>> RewriteValve does not function as expected. Besides it, there are also two
>>> https-443-exec executed by one request.
>>>
>>> There is definitely something wrong with Http2Protocol. And it results in
>>> this case a problem in RewriteValve.
>>>
>>> And what kind of extra information do you want to have. I cannot send any
>>> pictures in the email. You want the server.xml configuration? Except that
>>> file, almost every other configuration file is untouched. No load balancer,
>>> no any proxy here. Just a straight forward connection to tomcat instance.
>>>
>>> Here an remark, you can only reproduce errors if Http2Protocol is indeed
>>> activated, namely you see in browser h2 protocol is used by request. In
>>> Eclipse, I cannot reproduce the issue, since I cannot get h2 protocol
>>> working there.
>>>
>>>
>>> Hua
>>>
>>> Rémy Maucherat 于 2020年2月6日周四 下午3:06写道：
>>>
On Thu, Feb 6, 2020 at 2:56 PM Hua Zhang wrote:
> Thank you for the response. I am finally able to confirm the issue.
>
> When I put the following line in comment, everything works fine. I mean
as
> expected.
>
> **
>
>
> If the above line UpgradeProtocol is activated, I observe now at least
two
> weird situations.
>
> 1) As mentioned before, RewriteValve does not work as expected.
> 2) Besides it, I observed that serviet behaviors weird. According to
>>> the
> log file it seems that a servlet is sometimes called *twice by one
> request*.
>
> This is a snapshot of my log files. You can see that two https-443-exec
are
> called almost at the same time.
>
> 06-Feb-2020 13:38:04.676 SEVERE *[https-openssl-apr-443-exec-9]*
> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> for servlet [uploadServlet] in context with path [] threw exception
>
> org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
> Stream ended unexpectedly
> 06-Feb-2020 13:38:04.999 SEVERE *[https-openssl-apr-443-exec-2]*
> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> for servlet [uploadServlet] in context with path [] threw exception
>
> org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
> Stream ended unexpectedly
>
Ok, so your problem seems to be about HTTP/2 rather than the rewrite
>>> valve:
HTTP/2 is usually not used without TLS and ALPN. You should continue to
post more details. Something you can try is use the NIO connector (with
OpenSSL or JSSE) rather than APR, it would be a more common
>>> configuration.
Rémy
>
>
> Best regards,
>
> Hua
>>>
>>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Installing a program designed for Tomcat 5.5 on Tomcat 9
On 06/02/2020 20:32, Shane Johnson wrote: > I am currently trying to install a program designed to operate on Win XP 32 > and earlier on to a Win 10 environment. The program extracts to the Shared > and Webapps folders of Tomcat 5.5 and uses a SQL database. After converting > the database and installing it on SQL 2017 I added the JDBC connector and > downloaded and installed tomcat 9 only to find there is no shared folder to > extract the shared files to. Any suggestions? There are several options but the simplest is put anything extracted to shared in the lib directory. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RewriteValve does not work on HTTPS
The issue appears to be that the following RewriteCond does not match
when the request is served over HTTP/2 (with TLS) but it does if the
request is served over HTTP/1.1 with TLS.
RewriteCond %{HTTP_HOST} localhost
(Note the RewriteCond quoted at the start of this thread is invalid)
I'm wondering if this is related to the thread about
HttpServletRequest.getRemoteAddr() returning null at the start of a
request. It looks as if the request information may be being populated
too late.
Mark
On 06/02/2020 20:42, Pierre Chiu wrote:
> I just want to concur I have the same issue.
> Removing that one line upgradeprotocol
> and rewrite works again.
>
>
>
> On Thu, Feb 6, 2020 at 3:28 PM Hua Zhang wrote:
>
>> Nope, I also tested it with NIO instead of APR, same issues happen.
>> RewriteValve does not function as expected. Besides it, there are also two
>> https-443-exec executed by one request.
>>
>> There is definitely something wrong with Http2Protocol. And it results in
>> this case a problem in RewriteValve.
>>
>> And what kind of extra information do you want to have. I cannot send any
>> pictures in the email. You want the server.xml configuration? Except that
>> file, almost every other configuration file is untouched. No load balancer,
>> no any proxy here. Just a straight forward connection to tomcat instance.
>>
>> Here an remark, you can only reproduce errors if Http2Protocol is indeed
>> activated, namely you see in browser h2 protocol is used by request. In
>> Eclipse, I cannot reproduce the issue, since I cannot get h2 protocol
>> working there.
>>
>>
>> Hua
>>
>> Rémy Maucherat 于 2020年2月6日周四 下午3:06写道：
>>
>>> On Thu, Feb 6, 2020 at 2:56 PM Hua Zhang wrote:
>>>
Thank you for the response. I am finally able to confirm the issue.
When I put the following line in comment, everything works fine. I mean
>>> as
expected.
**
If the above line UpgradeProtocol is activated, I observe now at least
>>> two
weird situations.
1) As mentioned before, RewriteValve does not work as expected.
2) Besides it, I observed that serviet behaviors weird. According to
>> the
log file it seems that a servlet is sometimes called *twice by one
request*.
This is a snapshot of my log files. You can see that two https-443-exec
>>> are
called almost at the same time.
06-Feb-2020 13:38:04.676 SEVERE *[https-openssl-apr-443-exec-9]*
org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
for servlet [uploadServlet] in context with path [] threw exception
org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
Stream ended unexpectedly
06-Feb-2020 13:38:04.999 SEVERE *[https-openssl-apr-443-exec-2]*
org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
for servlet [uploadServlet] in context with path [] threw exception
org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
Stream ended unexpectedly
>>>
>>> Ok, so your problem seems to be about HTTP/2 rather than the rewrite
>> valve:
>>> HTTP/2 is usually not used without TLS and ALPN. You should continue to
>>> post more details. Something you can try is use the NIO connector (with
>>> OpenSSL or JSSE) rather than APR, it would be a more common
>> configuration.
>>>
>>> Rémy
>>>
Best regards,
Hua
>>>
>>
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RewriteValve does not work on HTTPS
I just want to concur I have the same issue. Removing that one line upgradeprotocol and rewrite works again. On Thu, Feb 6, 2020 at 3:28 PM Hua Zhang wrote: > Nope, I also tested it with NIO instead of APR, same issues happen. > RewriteValve does not function as expected. Besides it, there are also two > https-443-exec executed by one request. > > There is definitely something wrong with Http2Protocol. And it results in > this case a problem in RewriteValve. > > And what kind of extra information do you want to have. I cannot send any > pictures in the email. You want the server.xml configuration? Except that > file, almost every other configuration file is untouched. No load balancer, > no any proxy here. Just a straight forward connection to tomcat instance. > > Here an remark, you can only reproduce errors if Http2Protocol is indeed > activated, namely you see in browser h2 protocol is used by request. In > Eclipse, I cannot reproduce the issue, since I cannot get h2 protocol > working there. > > > Hua > > Rémy Maucherat 于 2020年2月6日周四 下午3:06写道： > > > On Thu, Feb 6, 2020 at 2:56 PM Hua Zhang wrote: > > > > > Thank you for the response. I am finally able to confirm the issue. > > > > > > When I put the following line in comment, everything works fine. I mean > > as > > > expected. > > > > > > ** > > > > > > > > > If the above line UpgradeProtocol is activated, I observe now at least > > two > > > weird situations. > > > > > > 1) As mentioned before, RewriteValve does not work as expected. > > > 2) Besides it, I observed that serviet behaviors weird. According to > the > > > log file it seems that a servlet is sometimes called *twice by one > > > request*. > > > > > > This is a snapshot of my log files. You can see that two https-443-exec > > are > > > called almost at the same time. > > > > > > 06-Feb-2020 13:38:04.676 SEVERE *[https-openssl-apr-443-exec-9]* > > > org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() > > > for servlet [uploadServlet] in context with path [] threw exception > > > > > > org.apache.commons.fileupload.MultipartStream$MalformedStreamException: > > > Stream ended unexpectedly > > > 06-Feb-2020 13:38:04.999 SEVERE *[https-openssl-apr-443-exec-2]* > > > org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() > > > for servlet [uploadServlet] in context with path [] threw exception > > > > > > org.apache.commons.fileupload.MultipartStream$MalformedStreamException: > > > Stream ended unexpectedly > > > > > > > Ok, so your problem seems to be about HTTP/2 rather than the rewrite > valve: > > HTTP/2 is usually not used without TLS and ALPN. You should continue to > > post more details. Something you can try is use the NIO connector (with > > OpenSSL or JSSE) rather than APR, it would be a more common > configuration. > > > > Rémy > > > > > > > > > > > Best regards, > > > > > > Hua > > >
Installing a program designed for Tomcat 5.5 on Tomcat 9
I am currently trying to install a program designed to operate on Win XP 32 and earlier on to a Win 10 environment. The program extracts to the Shared and Webapps folders of Tomcat 5.5 and uses a SQL database. After converting the database and installing it on SQL 2017 I added the JDBC connector and downloaded and installed tomcat 9 only to find there is no shared folder to extract the shared files to. Any suggestions?
Question on Apache Tomcat Patches
HI All- New to apache and apache patching. I have a question. When you run an apache patch (not an upgrade or install), should the patch create a new apache folder path, or simply modify & update the existing one? For example, if I am updating apache tomcat 7 to 7_99, should I expect to see a new folder C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7_99, or should I just see updates & modifications made to the existing folder C:\Program Files\Apache Software Foundation\Tomcat 7.0 ? Mike Walker W: 910.444.3018 M: 910.547.3387 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RewriteValve does not work on HTTPS
Nope, I also tested it with NIO instead of APR, same issues happen. RewriteValve does not function as expected. Besides it, there are also two https-443-exec executed by one request. There is definitely something wrong with Http2Protocol. And it results in this case a problem in RewriteValve. And what kind of extra information do you want to have. I cannot send any pictures in the email. You want the server.xml configuration? Except that file, almost every other configuration file is untouched. No load balancer, no any proxy here. Just a straight forward connection to tomcat instance. Here an remark, you can only reproduce errors if Http2Protocol is indeed activated, namely you see in browser h2 protocol is used by request. In Eclipse, I cannot reproduce the issue, since I cannot get h2 protocol working there. Hua Rémy Maucherat 于 2020年2月6日周四 下午3:06写道： > On Thu, Feb 6, 2020 at 2:56 PM Hua Zhang wrote: > > > Thank you for the response. I am finally able to confirm the issue. > > > > When I put the following line in comment, everything works fine. I mean > as > > expected. > > > > ** > > > > > > If the above line UpgradeProtocol is activated, I observe now at least > two > > weird situations. > > > > 1) As mentioned before, RewriteValve does not work as expected. > > 2) Besides it, I observed that serviet behaviors weird. According to the > > log file it seems that a servlet is sometimes called *twice by one > > request*. > > > > This is a snapshot of my log files. You can see that two https-443-exec > are > > called almost at the same time. > > > > 06-Feb-2020 13:38:04.676 SEVERE *[https-openssl-apr-443-exec-9]* > > org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() > > for servlet [uploadServlet] in context with path [] threw exception > > > > org.apache.commons.fileupload.MultipartStream$MalformedStreamException: > > Stream ended unexpectedly > > 06-Feb-2020 13:38:04.999 SEVERE *[https-openssl-apr-443-exec-2]* > > org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() > > for servlet [uploadServlet] in context with path [] threw exception > > > > org.apache.commons.fileupload.MultipartStream$MalformedStreamException: > > Stream ended unexpectedly > > > > Ok, so your problem seems to be about HTTP/2 rather than the rewrite valve: > HTTP/2 is usually not used without TLS and ALPN. You should continue to > post more details. Something you can try is use the NIO connector (with > OpenSSL or JSSE) rather than APR, it would be a more common configuration. > > Rémy > > > > > > > Best regards, > > > > Hua >
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Konstantin, please see below: You can configure a Valve for a specific web application by placing it into Context configuration for that specific web application (usually that is the "/META-INF/context.xml" configuration file). [1] [1] http://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Defining_a_context You are correct. I realized this after my original response. We actually already use per-context AccessLogValves. 2. If I understand correctly, the null value from request.getRemoteAddr() means that the client connection has already been closed. Tomcat cannot do much at that point, unless the information has not already been requested (and thus cached) when the connection was still alive. (The recent changes to the AccessLogValve are just that: to request the value earlier.) It is useless to process a request if the connection has already been closed. We request getRemoteAddr() several times during servlet processing (for Geolocation, proxy and carrier detection among other purposes), as well as in top-of-the-chain servlet filters. If the connection has been closed, it must be because of some Tomcat issue. This never happened with earlier Tomcar versions (we started this project many years ago with 6.x). Plus, the issue is only present in a small number of cases on HTTP/2, but not regular HTTPS nor plain HTTP. Why do you say that the null value is an invalid one? I do not see such words in the specification. I disagree. There cannot be a TCP/IP, HTTP, HTTPS, HTTP/2 or HTTP/3-QUIC connection without an originating IP, so it makes no sense to return NULL under any circumstances. Furthermore, the spec Javadoc for getRemoteAddr() states "Returns: a String containing the IP address of the client that sent the request" Contrast this to getRemoteUser() for instance, which states: "Returns: a String specifying the login of the user making this request, *or null if the user login is not known*" If NULL were a valid return value, it would be explicitly mentioned. 3. Just as a note (I would not recommend it for your specific use case) One known way to detect a closed connection is to trigger parameter parsing and look whether an error flag (implemented as an attribute of a Request) was set by it. See the implementation of org.apache.catalina.filters.FailedRequestFilter for an example. As a workaround, since we only have a very small number of requests that trigger this, we have implemented a filter that discards those requests and returns HTTP 400 "Bad Request", until the issue is resolved. 4. Do you run with the following configuration setting turned on? org.apache.catalina.connector.RECYCLE_FACADES=true We do not use this option, so we must be running with the default="false" See https://cwiki.apache.org/confluence/x/yColBg#TroubleshootingandDiagnostics-TroubleshootingunexpectedResponsestateproblems Thanks, I'll read through this. *Manuel Dominguez Sarmiento*
Re: [OT] distinction between resource charset and format octet decoding
On 2/6/2020 12:43 PM, Christopher Schultz wrote: … * Therefore `web.xml` settings, HTTP headers, etc. are all irrelevant, as this is an issue dealing with the file format itself, and the latest spec for the file format says to use UTF-8, so everyone should use UTF-8 already. Except for everyone who already uses something else and expects everything to be backward-compatible. I think there comes a time where we have to more forward after some critical level of usage is reached. I think we've passed that point. Modern browsers in the sense that you mention are not backwards-compatible for `application/x-www-form-urlencoded`. So what are we being compatible with by not using UTF-8 decoding? Do we have anything besides browsers consuming output from legacy JSP apps? As noted the browsers break when we try to be "backwards-compatible" in the sense you mention. The problem is that you don't get to declare what's "best" for everyone and then the whole world does what you want. But here I would imagine that already agrees what's best; the debate is whether we should do different than what we know is best because of some outdated specs. (And I say that as a huge proponent of following standards.) I'll give you an example that is directly relevant. Over 10 years ago I strongly advocated to the RDF group that the Internet should abandon the outdated practice of requiring that `text/*` media types default to US-ASCII; otherwise there would be no point in using `text/*` for anything going forward! (That's why we went through a sad phase where everyone was using `application/*` for text formats because they wanted to default to something other than US-ASCII.) * https://www.w3.org/2008/01/rdf-media-types * https://lists.w3.org/Archives/Public/www-archive/2007Dec/0059.html Sure enough, eventually someone saw the light (I won't claim I had anything to do with it, but it is exactly what I was arguing for) and created https://tools.ietf.org/html/rfc6657, which says that individual `text/*` types can choose a default other than ASCII. Finally we're not stuck in the past anymore! I would say that someone needs to create an updated `application/x-www-form-urlencoded` specification prescribing UTF-8 decoding of encoded octets, except that the WhatWG has already done that! So I'm not declaring that everyone should do it "my" way. I'm saying everyone should follow the latest spec which already exists. Anyway, thanks for listening. I think it's a fun discussion, and I wasn't being combative---I just wanted to tell a bit of the story. I need to get back to work now. :) Thanks again for the change in Tomcat 10! Garret
Re: [OT] distinction between resource charset and format octet decoding
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Garret, On 2/6/20 10:25 AM, Garret Wilson wrote: > On 2/6/2020 11:46 AM, André Warnier (tomcat/perl) wrote: >> … >>> As of Tomcat 10, conf/web.xml contains the following: >>> >>> >>> UTF-8 >>> UTF-8 >>> >>> >>> That *should* have the effect you are looking for but I confess I >>> haven't tested it in any great detail. >>> >> >> As I am sure many people (Christopher included) would agree, the >> real solution would be for browsers and other HTTP clients to >> indicate clearly in the request, the charset/encoding of each >> text parameter that they are sending. There are even HTTP headers >> already defined for that. > > > Which HTTP headers are you referring to? `Content-Type`? It is my > opinion that this is irrelevant and not applicable. > > As I explained (extensively) in my original post for this thread > back on 2019-01-08, the issue is not the charset of > `application/x-www-form-urlencoded`. That media type is made up of > ASCII characters. It doesn't matter whether you say it's ASCII, > ISO-8859-1, UTF-8, or whatever, the actual characters stay 100% the > same. Hmm. Not always. While it may be true that: 1. ASCII, ISO-8859-1, and UTF-8 are very common 2. ASCII, ISO-8859-1, and UTF-8 share the first 127 code points It is not true that: 3. All character encodings share the first 127 code points. UTF-16 doesn't follow that pattern. > At issue is when certain octets are encoded (as specified by the > `application/x-www-form-urlencoded` media type itself), what > charset to use when decoding them. This is independent of the > encoding of the media type itself; rather this is defined by the > specification for the format. Correct. And there is lack of agreement for URLs, so browsers decided to make it up. It's not possible to guess what the browser has chosen because it does not advertise it in any way (absent a standard). The only 100% reliable way to do it would be to add a parameter to every request which has a known-correct value that can be unambiguously decoded. You just keep re-decoding the whole URL until that parameter value matches the known-correct value. Sounds like a lot of fun to implement across a whole application, right? > Unfortunately https://tools.ietf.org/html/rfc1866 actually says we > should use ASCII when decoding the octets, but this is severely > antiquated and doesn't fit with modern practice. The WhatWG > essentially redefines the format to say that the octets must be > interpreted as UTF-8: > > https://url.spec.whatwg.org/#application/x-www-form-urlencoded > > So to summarize my view: > > * The decoding of the `application/x-www-form-urlencoded` media > type encoded octets is completely independent of the charset > indicated in the `Content-Type` header, and rather goes to the > specification of the format itself. It's strange, because Content-Type can contain a charset parameter, but MIME specifically says that "charset" parameters are only appropriate for "text/*" MIME types. So for application/x-www-form-urlencoded, you "shouldn't" add that parameter. But there's no particular reason NOT to include it (it doesn't actually violate any spec) and adding it COMPLETELY AND UNAMBIGUOUSLY indicates what the browser chose as the encoding. > * RFC 1866 is severely out of date and out of step, and the > WhatWG's specification of the `application/x-www-form-urlencoded` > media type should be used instead. (Modern browser practice would > seem to agree with me.) RFC 1886 has been very much superseded. Also, HTML specs shouldn't be defining HTTP semantics. So ignore whatever is in RFC 1866 on multiple grounds. > * Therefore `web.xml` settings, HTTP headers, etc. are all > irrelevant, as this is an issue dealing with the file format > itself, and the latest spec for the file format says to use UTF-8, > so everyone should use UTF-8 already. Except for everyone who already uses something else and expects everything to be backward-compatible. The problem is that you don't get to declare what's "best" for everyone and then the whole world does what you want. I happen to agree with you (Everyone should move to UTF-8 for everything. Everywhere. Forever.), but you have to recognize that there is history and entrenched systems, environments, and mindsets. > The new default `web.xml` in Tomcat 10 is a wonderful step in the > right direction. +1 - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl48NAoACgkQHPApP6U8 pFgJ6A/+JSArcUkqm3P6n0awICXTuqIx0TU1oIf9bzivpAI/Na9fr//ebnwzmvoy EXpbnn97B7Sy8uZ1wvT0+PQLbmwVmM/f7zBk4q+7Ba/ogkmrSHeLlsCIbLAlXOLD kr/xDE4ftxrwR2+ZwuQwxH0muFH+4rq2SBFWTQnGORCQDqRRK7eQoQYHWE0HIAxj cAJmwkQEQyi+YHdgaUo0L4BU7lvgPGk7JyjbzWBiigFYy/1Du1caE7PzYLa5G3wZ BrYDA6QoQA+nUmXHn/ayUVXvsZc2l/nU/uM5m68Tp1iEVxdgp4u8XtHuqgv0Nzda IeQq9HOP8wd7l27/dk2DvlZBmSWt2XDOI5ig+NoLPT1ixyQIqVJ2K8SyayGdUHW9
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Manuel,
On 2/5/20 1:29 PM, Manuel Dominguez Sarmiento wrote:
> Yes, there are two reasons:
>
> 1) The Tomcat valves operate on all webapps. We only need/require
> this for one particular webapp without affecting the others.
Not true; see Konstantin's response.
> 2) The code has been simplified for illustration purposes. Besides
> X-Forwarded-For, we detect and work around many other custom
> external mobile proxies which do not use X-Forwarded-For and
> require custom Geolocation code to detect the ISP and connection
> type (Google Compression Proxy, Nokia OVI, Novarra, Lotus Flare,
> Opera Mini, Opera Max, Samsung Max, etc.) - this kind of
> customization is not possible without custom code.
Interesting. Is this something you think would be widely useful and/or
would be willing to share with the community? If it's a fast-moving
target (e.g. new public proxies are popping-up all the time, or
existing proxies keep changing their configuration requirements) then
maybe it's not a great fit for a stable product like Tomcat.
On the other hand, if it could be configured relatively easily (like
with a "proxy definitions" file or something), then it could still be
very valuable even with a simple or default configuration which only
supports some very large proxies (e.g. Akamai, CloudFront, CloudFlare,
etc.).
> This filter is not meant for detecting internal proxies within our
> control (such as Apache front ends or load balancers), but rather
> public proxies which are "transparently" (not really) used via
> some mobile devices and services.
Does it matter whether these are "internal" versus "external" proxies?
The only real difference is the IP-range of the proxy, right? It
doesn't matter whether you control the proxy or it's an external
service: you still have to secure and validate the connection in the
same ways, and take the same action(s) on the server-end where you
trust the information being presented.
- -chris
> On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote:
Our filter is not doing anything fancy (and it has always
worked correctly before we ran into this bug). In
pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr(); boolean isProxy =
isProxy(ip); if (isProxy) { String unwrappedIP =
unwrapXForwardedFor(request); chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request,
unwrappedIP), response); } else { chain.doFilter(request,
response); } }
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating
to the actual request, while unwrapXForwardedFor() does what
the name suggests, which is processing X-Forwarded-For to
obtain the originating IP before it hit the detected proxy.
> Any reason not to use the valves Tomcat provides to do pretty much
> this exact thing?
>
> https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_V
al
>
>
ve
>
> -chris
>>
>> -
>>
>>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl48MBsACgkQHPApP6U8
pFgaXw/+P0rEqNNqW2oM3Yajrzk3mgYt41kuq2zGrjdr/6vBGR1fUmrx92krOkM/
1bzFOkbWnRWIktFmjyARaGK12F3/zczvU7JRmRazEKXraxJ0vgPg8NPy8YN4KhBG
efptbwqbbdJ02r1i8eVdNjacYBd5/gT51qFRaaTseIz16prSUxaT4RXoui2je68v
fkA7pl5jAND8B7Nr6uczGVQwQELWTEwNKUiz6ji+GAwKF7oZMCX64p7TYEcD7o2p
BgcO5VzyjwdcXuDTOJ4RqTgaHc09aqdP+VRiWV18RGr0rYTISq6zW6lHXT9Goc0m
qH5mKp402aiX03rK8n9F+523K52X7xR7B0+48r+4UcPDrWiWDzuF/IVI93ugNLw5
ITexNxhvmhvC2PHY5CRoa69Us2YG9iZo0z579RIFuYT4/75DR3+oakqIAHz2FU5C
x+n4Qkbaj/v/O56Ja983bKbWul5XCTLnL2AmLKfcnK7er1CpWx9elN7oXvhiFYC1
8ceLtQU1A36vElFWUj1jC82M41tuQdcdL3VADcvi6bANpAYEsEG/RbNnBu/0WgfD
ByxN40M9Km+wtzDlxqW5Sg2eKm4tdNk2duqtTv9+NPH4H+5Tu9+OQ/r5gTPrsJfn
knEd9/Los4QCrAYSsHF6PXD/bbdlJbUpRdIMYUKJXA6BNMT2LNo=
=1ts9
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] distinction between resource charset and format octet decoding
On 2/6/2020 11:46 AM, André Warnier (tomcat/perl) wrote: … As of Tomcat 10, conf/web.xml contains the following: UTF-8 UTF-8 That *should* have the effect you are looking for but I confess I haven't tested it in any great detail. As I am sure many people (Christopher included) would agree, the real solution would be for browsers and other HTTP clients to indicate clearly in the request, the charset/encoding of each text parameter that they are sending. There are even HTTP headers already defined for that. Which HTTP headers are you referring to? `Content-Type`? It is my opinion that this is irrelevant and not applicable. As I explained (extensively) in my original post for this thread back on 2019-01-08, the issue is not the charset of `application/x-www-form-urlencoded`. That media type is made up of ASCII characters. It doesn't matter whether you say it's ASCII, ISO-8859-1, UTF-8, or whatever, the actual characters stay 100% the same. At issue is when certain octets are encoded (as specified by the `application/x-www-form-urlencoded` media type itself), what charset to use when decoding them. This is independent of the encoding of the media type itself; rather this is defined by the specification for the format. Unfortunately https://tools.ietf.org/html/rfc1866 actually says we should use ASCII when decoding the octets, but this is severely antiquated and doesn't fit with modern practice. The WhatWG essentially redefines the format to say that the octets must be interpreted as UTF-8: https://url.spec.whatwg.org/#application/x-www-form-urlencoded So to summarize my view: * The decoding of the `application/x-www-form-urlencoded` media type encoded octets is completely independent of the charset indicated in the `Content-Type` header, and rather goes to the specification of the format itself. * RFC 1866 is severely out of date and out of step, and the WhatWG's specification of the `application/x-www-form-urlencoded` media type should be used instead. (Modern browser practice would seem to agree with me.) * Therefore `web.xml` settings, HTTP headers, etc. are all irrelevant, as this is an issue dealing with the file format itself, and the latest spec for the file format says to use UTF-8, so everyone should use UTF-8 already. The new default `web.xml` in Tomcat 10 is a wonderful step in the right direction. See my original post for more in-depth explanation. Garret
Re: [OT] distinction between resource charset and format octet decoding
On 06.02.2020 14:44, Mark Thomas wrote: On 06/02/2020 13:39, Garret Wilson wrote: On 2/6/2020 10:36 AM, Mark Thomas wrote: … Whether Tomcat should ship with this setting present in conf/web.xml by default is something that should probably be discussed for Tomcat 10. Given the current state of the web, there is a reasonable case for doing so. I'll add that to the TOMCAT-NEXT discussion list. Is this still on the list for discussion for Tomcat 10? No, because it has already been implemented for Tomcat 10 and is in the milestone release currently being voted on. Waitasec. I'm not used to good news, so I want to make sure I understand what you're saying. Are you saying that the proposed Tomcat 10 implementation already interprets encoded octets in web form submissions using UTF-8 by default?!! :O As of Tomcat 10, conf/web.xml contains the following: UTF-8 UTF-8 That *should* have the effect you are looking for but I confess I haven't tested it in any great detail. As I am sure many people (Christopher included) would agree, the real solution would be for browsers and other HTTP clients to indicate clearly in the request, the charset/encoding of each text parameter that they are sending. There are even HTTP headers already defined for that. (Nowadays the default could be Unicode/UTF-8). The problem is that browsers and other agents don't do that, although they undoubtedly always know themselves, and although it would solve a series of issues that have literally been there forever at the server and application level (*). I have often wondered if/why the Apache Foundation does not pack enough influence over the HTTP/HTML specifications process and over browser producers, to achieve that. (And if not the Apache Foundation, then who ?) (*) My own guess is that this basic thing (or lack of it) has cost over the years many thousands of lines of unnecessary code and many thousands of unproductive developer hours. As a tiny example, just consider the above web.xml parameters, and how much time in total was dedicated to their definition and implementation.. Never mind all the previous related filters and valves and their discussions on this list. And that's only for Tomcat. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RewriteValve does not work on HTTPS
On Thu, Feb 6, 2020 at 2:56 PM Hua Zhang wrote:
> Thank you for the response. I am finally able to confirm the issue.
>
> When I put the following line in comment, everything works fine. I mean as
> expected.
>
> **
>
>
> If the above line UpgradeProtocol is activated, I observe now at least two
> weird situations.
>
> 1) As mentioned before, RewriteValve does not work as expected.
> 2) Besides it, I observed that serviet behaviors weird. According to the
> log file it seems that a servlet is sometimes called *twice by one
> request*.
>
> This is a snapshot of my log files. You can see that two https-443-exec are
> called almost at the same time.
>
> 06-Feb-2020 13:38:04.676 SEVERE *[https-openssl-apr-443-exec-9]*
> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> for servlet [uploadServlet] in context with path [] threw exception
>
> org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
> Stream ended unexpectedly
> 06-Feb-2020 13:38:04.999 SEVERE *[https-openssl-apr-443-exec-2]*
> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> for servlet [uploadServlet] in context with path [] threw exception
>
> org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
> Stream ended unexpectedly
>
Ok, so your problem seems to be about HTTP/2 rather than the rewrite valve:
HTTP/2 is usually not used without TLS and ALPN. You should continue to
post more details. Something you can try is use the NIO connector (with
OpenSSL or JSSE) rather than APR, it would be a more common configuration.
Rémy
>
>
> Best regards,
>
> Hua
>
>
> On Wed, Feb 5, 2020 at 2:15 PM Felix Schumacher <
> felix.schumac...@internetallee.de> wrote:
>
> > Am 04.02.2020 22:16, schrieb Hua Zhang:
> > > What I mean with word 'works' is: the RewriteRule has been executed.
> > >
> > > That is not the case by HTTPS. The rule has not been executed while the
> > > RewriteCond is fulfilled.
> >
> > Can you give us more information on your setup? Is there any
> > Proxy/Loadbalancer in front of your tomcat? If so, can you show us
> > details on that setup?
> > What is the value of the host request header in both cases?
> >
> > Felix
> > >
> > > Olaf Kock 于 2020年2月4日周二 下午9:06写道：
> > >
> > >>
> > >> On 04.02.20 20:31, Hua Zhang wrote:
> > >> > Best tomcat team,
> > >> >
> > >> > Hereby I have a question about an issue I found by using
> RewriteValve
> > >> > on tomcat 9.30
> > >> >
> > >> > The rewrite.config is very simple:
> > >> >
> > >> > /RewriteCond %{HTTP_HOST} =youkoop.com
> > >> > RewriteRule ^.*$ https://www.youkoop.com [R=301,L]
> > >> > /
> > >> >
> > >> > All I want is just redirect a naked root domain to a www domain with
> > >> > HTTPS.
> > >> >
> > >> > The redirection works on HTTP but not HTTPS.
> > >> >
> > >> > http://youkoop.com => https://www.youkoop.com *works*
> > >> >
> > >> Note: Images don't get through in this mailing list. I can imagine
> > >> what
> > >> "works" means, but for your next example: Please elaborate what "does
> > >> not work" means.
> > >> >
> > >> > *https*://youkoop.com =>
> > >> > https://www.youkoop.com *does not work*
> > >>
> > >> First thing to test: Does https://youkoop.com work without the
> > >> redirect,
> > >> then with the "wrong" host name? Otherwise it might be as simple as a
> > >> misconfigured TLS host that's never invoked because of a certificate
> > >> mismatch.
> > >>
> > >> Olaf
> > >>
> > >>
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
>
Re: RewriteValve does not work on HTTPS
Thank you for the response. I am finally able to confirm the issue.
When I put the following line in comment, everything works fine. I mean as
expected.
**
If the above line UpgradeProtocol is activated, I observe now at least two
weird situations.
1) As mentioned before, RewriteValve does not work as expected.
2) Besides it, I observed that serviet behaviors weird. According to the
log file it seems that a servlet is sometimes called *twice by one request*.
This is a snapshot of my log files. You can see that two https-443-exec are
called almost at the same time.
06-Feb-2020 13:38:04.676 SEVERE *[https-openssl-apr-443-exec-9]*
org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
for servlet [uploadServlet] in context with path [] threw exception
org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
Stream ended unexpectedly
06-Feb-2020 13:38:04.999 SEVERE *[https-openssl-apr-443-exec-2]*
org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
for servlet [uploadServlet] in context with path [] threw exception
org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
Stream ended unexpectedly
Best regards,
Hua
On Wed, Feb 5, 2020 at 2:15 PM Felix Schumacher <
felix.schumac...@internetallee.de> wrote:
> Am 04.02.2020 22:16, schrieb Hua Zhang:
> > What I mean with word 'works' is: the RewriteRule has been executed.
> >
> > That is not the case by HTTPS. The rule has not been executed while the
> > RewriteCond is fulfilled.
>
> Can you give us more information on your setup? Is there any
> Proxy/Loadbalancer in front of your tomcat? If so, can you show us
> details on that setup?
> What is the value of the host request header in both cases?
>
> Felix
> >
> > Olaf Kock 于 2020年2月4日周二 下午9:06写道：
> >
> >>
> >> On 04.02.20 20:31, Hua Zhang wrote:
> >> > Best tomcat team,
> >> >
> >> > Hereby I have a question about an issue I found by using RewriteValve
> >> > on tomcat 9.30
> >> >
> >> > The rewrite.config is very simple:
> >> >
> >> > /RewriteCond %{HTTP_HOST} =youkoop.com
> >> > RewriteRule ^.*$ https://www.youkoop.com [R=301,L]
> >> > /
> >> >
> >> > All I want is just redirect a naked root domain to a www domain with
> >> > HTTPS.
> >> >
> >> > The redirection works on HTTP but not HTTPS.
> >> >
> >> > http://youkoop.com => https://www.youkoop.com *works*
> >> >
> >> Note: Images don't get through in this mailing list. I can imagine
> >> what
> >> "works" means, but for your next example: Please elaborate what "does
> >> not work" means.
> >> >
> >> > *https*://youkoop.com =>
> >> > https://www.youkoop.com *does not work*
> >>
> >> First thing to test: Does https://youkoop.com work without the
> >> redirect,
> >> then with the "wrong" host name? Otherwise it might be as simple as a
> >> misconfigured TLS host that's never invoked because of a certificate
> >> mismatch.
> >>
> >> Olaf
> >>
> >>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
Re: distinction between resource charset and format octet decoding
On 2/6/2020 10:44 AM, Mark Thomas wrote: … As of Tomcat 10, conf/web.xml contains the following: UTF-8 UTF-8 That *should* have the effect you are looking for but I confess I haven't tested it in any great detail. Yes! Oh, that is so wonderful. Thank you! I brought this issue up on the list over a year ago, and I have since published my entire comprehensive software development course (still being expanded). https://www.globalmentor.com/courses/softdev/ The course is centered around Tomcat as the server, and the lesson on HTML forms contains a section warning to use ``. https://www.globalmentor.com/courses/softdev/html-forms Once Tomcat 10 is released I'll be able to update this note as well. Thanks again! Garret - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: distinction between resource charset and format octet decoding
On 06/02/2020 13:39, Garret Wilson wrote: > On 2/6/2020 10:36 AM, Mark Thomas wrote: >> … Whether Tomcat should ship with this setting present in conf/web.xml by default is something that should probably be discussed for Tomcat 10. Given the current state of the web, there is a reasonable case for doing so. I'll add that to the TOMCAT-NEXT discussion list. >>> Is this still on the list for discussion for Tomcat 10? >> No, because it has already been implemented for Tomcat 10 and is in the >> milestone release currently being voted on. > > Waitasec. I'm not used to good news, so I want to make sure I understand > what you're saying. Are you saying that the proposed Tomcat 10 > implementation already interprets encoded octets in web form submissions > using UTF-8 by default?!! :O As of Tomcat 10, conf/web.xml contains the following: UTF-8 UTF-8 That *should* have the effect you are looking for but I confess I haven't tested it in any great detail. Mark > > It will be a joy to update the FAQ when this is released. > > Garret > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: distinction between resource charset and format octet decoding
On 2/6/2020 10:36 AM, Mark Thomas wrote: … Whether Tomcat should ship with this setting present in conf/web.xml by default is something that should probably be discussed for Tomcat 10. Given the current state of the web, there is a reasonable case for doing so. I'll add that to the TOMCAT-NEXT discussion list. Is this still on the list for discussion for Tomcat 10? No, because it has already been implemented for Tomcat 10 and is in the milestone release currently being voted on. Waitasec. I'm not used to good news, so I want to make sure I understand what you're saying. Are you saying that the proposed Tomcat 10 implementation already interprets encoded octets in web form submissions using UTF-8 by default?!! :O It will be a joy to update the FAQ when this is released. Garret - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: distinction between resource charset and format octet decoding
On 06/02/2020 13:30, Garret Wilson wrote: > On 1/8/2019 9:57 PM, Mark Thomas wrote: >> … >> >> Yes, this default is now very out-dated. That is a side-effect of: >> … >> As of Servlet 4.0 there is a specification compliant configuration >> option to change this default to any encoding of your choice. >> Obviously, UTF-8 is one of the options. You can do this by adding the >> following to your web.xml: >> … >> >> Whether Tomcat should ship with this setting present in conf/web.xml >> by default is something that should probably be discussed for Tomcat >> 10. Given the current state of the web, there is a reasonable case for >> doing so. I'll add that to the TOMCAT-NEXT discussion list. > > Is this still on the list for discussion for Tomcat 10? No, because it has already been implemented for Tomcat 10 and is in the milestone release currently being voted on. Mark > > In my opinion it would be a real shame if Tomcat 10 ships with a web > form encoding default that goes against the WhatWG specifications and > corrupts non ISO-8859-1 content under modern browsers. > > Garret > > P.S. Mark, please ignore the other email from my personal email address. > Because the Tomcat users list doesn't include my name in the "To:" > header, my email client didn't know to use the correct reply address. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: distinction between resource charset and format octet decoding
On 1/8/2019 9:57 PM, Mark Thomas wrote: … Yes, this default is now very out-dated. That is a side-effect of: … As of Servlet 4.0 there is a specification compliant configuration option to change this default to any encoding of your choice. Obviously, UTF-8 is one of the options. You can do this by adding the following to your web.xml: … Whether Tomcat should ship with this setting present in conf/web.xml by default is something that should probably be discussed for Tomcat 10. Given the current state of the web, there is a reasonable case for doing so. I'll add that to the TOMCAT-NEXT discussion list. Is this still on the list for discussion for Tomcat 10? In my opinion it would be a real shame if Tomcat 10 ships with a web form encoding default that goes against the WhatWG specifications and corrupts non ISO-8859-1 content under modern browsers. Garret P.S. Mark, please ignore the other email from my personal email address. Because the Tomcat users list doesn't include my name in the "To:" header, my email client didn't know to use the correct reply address. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
ср, 5 февр. 2020 г. в 21:29, Manuel Dominguez Sarmiento : > > Yes, there are two reasons: > > 1) The Tomcat valves operate on all webapps. We only need/require this > for one particular webapp without affecting the others. You can configure a Valve for a specific web application by placing it into Context configuration for that specific web application (usually that is the "/META-INF/context.xml" configuration file). [1] [1] http://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Defining_a_context 2. If I understand correctly, the null value from request.getRemoteAddr() means that the client connection has already been closed. Tomcat cannot do much at that point, unless the information has not already been requested (and thus cached) when the connection was still alive. (The recent changes to the AccessLogValve are just that: to request the value earlier.) It is useless to process a request if the connection has already been closed. Why do you say that the null value is an invalid one? I do not see such words in the specification. 3. Just as a note (I would not recommend it for your specific use case) One known way to detect a closed connection is to trigger parameter parsing and look whether an error flag (implemented as an attribute of a Request) was set by it. See the implementation of org.apache.catalina.filters.FailedRequestFilter for an example. 4. Do you run with the following configuration setting turned on? org.apache.catalina.connector.RECYCLE_FACADES=true See https://cwiki.apache.org/confluence/x/yColBg#TroubleshootingandDiagnostics-TroubleshootingunexpectedResponsestateproblems Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
