Re: Parallel deploy with /manager/text servlet
Thanks Remy, I now see that Mark has already been working on this and
committed 7 patches to resolve this issue. I see that these commits were
made right after the release of 9.0.43 (Feb. 1st while 9.0.43 was
release on Jan. 28th)
https://github.com/apache/tomcat/commit/ee6069392752f930c64e160a6a08a9904ffea008
I assume these fixes will be available on 9.0.44 right? I'm unsure
whether they were just applied to the 10.x branch or backported to the
9.x branch as well.
Thanks
*Manuel Dominguez Sarmiento*
On 11/02/2021 10:11, Manuel Dominguez Sarmiento wrote:
Sorry it took so long to get back to this. @Mark I've just filed bug
65136 as requested:
https://bz.apache.org/bugzilla/show_bug.cgi?id=65136
*Manuel Dominguez Sarmiento*
On 07/01/2021 05:40, Mark Thomas wrote:
On 06/01/2021 21:48, Manuel Dominguez Sarmiento wrote:
Hi, our system consists on about 80+ webapps on different servers.
Multiple webapps are deployed per Tomcat server remotely using the
manager and host-manager webapps.
This has been working fine for years, however some servers now have 5+
webapps and they are taking a long time to start. We invoke the deploy
command on the manager webapp in parallel for each webapp, in order to
attempt a reduction in total server startup time. However the speedup is
very modest. The servers have powerful Xeon multi-core servers so a
parallel speedup should be possible.
Looking at the source code for
org.apache.catalina.manager.ManagerServlet we see that the following
method is synchronized which would explain the behaviour we are seeing:
protected synchronized void deploy
(PrintWriter writer, String config, ContextName cn,
String tag, boolean update, HttpServletRequest request,
StringManager smClient) {
Is there any real reason why this needs to be synchronized? Tomcat can
deploy webapps in parallel when backgroundProcessorDelay is set to a
positive number, but this is not useful when remotely deploying webapps
using the manager webapp.
Any ideas? Thanks
Some svn archaeology uncovered this commit comment:
"Known issue: locking occurs on an uploaded WAR, for reasons which
elude me right now."
The method level sync looks like an attempt to ensure that there aren't
parallel attempts to deploy a web application to the same path. I can't
think of any other reason. But locking on, essentially, the virtual
host, seems overly broad. A lock on the deployment path would be sufficient.
There are also calls to isServiced(), addServiced() and removeServiced()
which look to be trying to do the same thing which have their own issues
(not atomic, uses a List rather than a Set).
We do need to ensure that we don't get parallel attempts to service the
same web application. There is scope for all sorts of bad things to
happen if that is possible.
Overall, I think this can be fixed but it looks like some API changes
are going to be required which makes things a little more complicated.
It would be worth you opening a Bugzilla issue for this to make sure it
doesn't get forgotten about. The issues with the unsynchronized
deployment methods are sufficient - in my view - to class this as a bug
rather than an enhancement.
Mark
-
To unsubscribe, e-mail:users-unsubscr...@tomcat.apache.org
For additional commands, e-mail:users-h...@tomcat.apache.org
Re: Parallel deploy with /manager/text servlet
Sorry it took so long to get back to this. @Mark I've just filed bug
65136 as requested:
https://bz.apache.org/bugzilla/show_bug.cgi?id=65136
*Manuel Dominguez Sarmiento*
On 07/01/2021 05:40, Mark Thomas wrote:
On 06/01/2021 21:48, Manuel Dominguez Sarmiento wrote:
Hi, our system consists on about 80+ webapps on different servers.
Multiple webapps are deployed per Tomcat server remotely using the
manager and host-manager webapps.
This has been working fine for years, however some servers now have 5+
webapps and they are taking a long time to start. We invoke the deploy
command on the manager webapp in parallel for each webapp, in order to
attempt a reduction in total server startup time. However the speedup is
very modest. The servers have powerful Xeon multi-core servers so a
parallel speedup should be possible.
Looking at the source code for
org.apache.catalina.manager.ManagerServlet we see that the following
method is synchronized which would explain the behaviour we are seeing:
protected synchronized void deploy
(PrintWriter writer, String config, ContextName cn,
String tag, boolean update, HttpServletRequest request,
StringManager smClient) {
Is there any real reason why this needs to be synchronized? Tomcat can
deploy webapps in parallel when backgroundProcessorDelay is set to a
positive number, but this is not useful when remotely deploying webapps
using the manager webapp.
Any ideas? Thanks
Some svn archaeology uncovered this commit comment:
"Known issue: locking occurs on an uploaded WAR, for reasons which
elude me right now."
The method level sync looks like an attempt to ensure that there aren't
parallel attempts to deploy a web application to the same path. I can't
think of any other reason. But locking on, essentially, the virtual
host, seems overly broad. A lock on the deployment path would be sufficient.
There are also calls to isServiced(), addServiced() and removeServiced()
which look to be trying to do the same thing which have their own issues
(not atomic, uses a List rather than a Set).
We do need to ensure that we don't get parallel attempts to service the
same web application. There is scope for all sorts of bad things to
happen if that is possible.
Overall, I think this can be fixed but it looks like some API changes
are going to be required which makes things a little more complicated.
It would be worth you opening a Bugzilla issue for this to make sure it
doesn't get forgotten about. The issues with the unsynchronized
deployment methods are sufficient - in my view - to class this as a bug
rather than an enhancement.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Parallel deploy with /manager/text servlet
Hi, our system consists on about 80+ webapps on different servers.
Multiple webapps are deployed per Tomcat server remotely using the
manager and host-manager webapps.
This has been working fine for years, however some servers now have 5+
webapps and they are taking a long time to start. We invoke the deploy
command on the manager webapp in parallel for each webapp, in order to
attempt a reduction in total server startup time. However the speedup is
very modest. The servers have powerful Xeon multi-core servers so a
parallel speedup should be possible.
Looking at the source code for
org.apache.catalina.manager.ManagerServlet we see that the following
method is synchronized which would explain the behaviour we are seeing:
protected synchronized void deploy
(PrintWriter writer, String config, ContextName cn,
String tag, boolean update, HttpServletRequest request,
StringManager smClient) {
Is there any real reason why this needs to be synchronized? Tomcat can
deploy webapps in parallel when backgroundProcessorDelay is set to a
positive number, but this is not useful when remotely deploying webapps
using the manager webapp.
Any ideas? Thanks
*Manuel Dominguez Sarmiento*
Re: java.lang.OutOfMemoryError: PermGen space when we redeploy same application multiple times
The articles in this page will be helpful: https://java.jiderhamn.se/category/classloader-leaks/ On 12/10/2020 04:19, Mark Thomas wrote: On 11/10/2020 02:39, Prabhu Gurunathan wrote: Hi All, We have an setup where we are using OpenJDK 1.7 and Tomcat 7.0.100 , in CentOs 7 Env . and we have many application deployed in Tomcat/webapps and the common lib's needed for those apps are kept in Tomcat/lib directory like log4j , commons-fileupload ,xerces , Xalan .. etc The problem here is When we try to undeploy and deploy same applications multiple time we are getting into java.lang.OutOfMemoryError: PermGen space very quickly . Want to know is it very generic problem on this deployment model or is this can be fixed anyway ? The memory leak could be in any of: - the application code - a library the application depends on - the JVM - Tomcat In all cases, it should be possible to fix it. There might even be a short-term workaround available. First of all, make sure that the JreMemoryLeakPreventionListener is enabled in your configuration. Secondly, make sure that you don't get any warnings about possible memory leaks in the logs when you reload an application. If you do, fix the leaks identified. If you still see issues, the short version is: - user a profiler - reload each app in turn until you see more strong references to org.apache.catalina.loader.[Parallel]WebappClassLoader instances in memory than you have web applications - find the one where started = false - trace its GC roots - that will tell you where the memory leak is - how it was created might be harder to track down The long version is in a presentation linked from the Tomcat web site: https://home.apache.org/~markt/presentations/2010-08-05-Memory-Leaks-JavaOne-60mins.pdf If you have any questions, you can ask here. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JMX Insecure Agent.
JMX is usually setup on port 1099 for monitoring the JVM. It can be either secured, or insecure (no password, no encryption) which is the default configuration. If you cannot modify the app, then the safest bet would probably be to block access to the port with the system firewall (for instance, iptables on Linux). Check the following system properties for clues: -Dcom.sun.management.jmxremote.port= -Dcom.sun.management.jmxremote.password.file= -Dcom.sun.management.jmxremote.access.file= - Manuel Dominguez Sarmiento On 06/08/2020 10:13, Kaydo Bramble wrote: Hi Everyone, Our security scanner has identified an application that has "Java JMX Agent Insecure Configuration" on one of our Tomcat 8.5 servers. This server was setup by a vendor and I am not sure what JMX is being used for or how it is setup. Does anyone have any ideas on how to resolve this? I tried asking the vendor multiple times and they have no clue since 2019. Thanks, Kenrick "Kaydo" Bramble Manager, Databases and Middleware - Enterprise Systems Office of Information Technology <mailto:ka...@rice.edu> ka...@rice.edu | 713-348-8645 Rice University | 6100 Main St. | Houston, TX 77005 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Should Tomcat 10 enable response compression by default?
I would not change this default. GZIP (or other kinds) of response compression are better addressed as servlet filters. Having the Tomcat feature is good, but IMHO it should only be enabled by those who need it. At least in our case we have our own code to deal with this, considering proxying, CDN, buggy browsers, etc. *Manuel Dominguez Sarmiento* On 09/06/2020 17:20, Mark Thomas wrote: Hi all, An enhancement has been opened to enable response compression by default: https://bz.apache.org/bugzilla/show_bug.cgi?id=64431 In short, the proposal is to change the default for the Connector's compression attribute from "off" to "on". This would be for Tomcat 10 onwards only. The following would be unchanged: - compressibleMimeType - compressionMinSize - noCompressionStrongETag It would be helpful to know what the range of views of the user community are on this proposal. So, thoughts? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Http2 tomact server taking time in responding when 1st StreamId is a large integer value like 2147483641
I must say that we're also seeing weird, seemingly random response delays from Tomcat on HTTP/2 We haven't looked into it at such a low level though. We're currently on 9.0.35 but we've been seeing this on previous versions as well. *Manuel Dominguez Sarmiento* On 21/05/2020 05:32, Prateek Kohli wrote: Hello, Tomcat version : 9.0.29 We are running a Tomcat Http2 Server and a Jetty http2 client. When we send 1^st request from Jetty client to tomcat server with streamId number as 1, tomcat sends the WINDOW_UPDATE header and the response in 1~2 milliseconds. Packet number 164 is the response in the below tcpdump. But when we send the 1st request from jetty client to tomcat server with streamId as 2147483641, the 1^st response from tomcat comes after almost 5 seconds And the response for subsequent requests comes within 1~2 milliseconds. In the below tcpdump it can be seen that the response packet number 167 comes after almost 5 seconds from the tomcat server. Would you please be able to explain why the response from tomcat server is getting delayed when the 1^st StreamId number is a large integer i.e. 2147483641. Regards, Prateek Kohli
Re: Large file uploads via https fail on Tomcat 9
Glad that helped!* Manuel Dominguez Sarmiento* On 24/04/2020 06:42, OIT Nua wrote: Thanks, again, for the assistance. Upgrading to 9.0.34 has resolved the issue.
Re: Large file uploads via https fail on Tomcat 9
See the following thread: http://tomcat.10.x6.nabble.com/Uploads-breaking-post-upgrade-to-9-0-31-td5096655.html And this bug report: https://bz.apache.org/bugzilla/show_bug.cgi?id=64202 It looks as if upgrading to 9.0.34 should resolve this. On 21/04/2020 14:55, OIT Nua wrote: Thanks for responding. We are Tomcat version 9.0.31, running on Windows 2012 R2 Standard. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Large file uploads via https fail on Tomcat 9
What version/build are you using? I believe the was a recent bug that has been already resolved regarding this on HTTPS. I have a Tomcat application that accepts file uploads via https. I have had no problems with file uploads and https in Tomcat versions 7 and 8. Now that we've ported to Tomcat 9, file uploads larger than approx. 173KB just hang and never complete (in fact, they seem from the Tomcat side to never start; from the user side, it looks like the upload is progressing, but it just hangs and never completes). Small (<173KB) files upload via https fine. I have adjusted various settings, including disableUPloadTimeout, connectionUploadTimeout, maxFileSize, etc., with nothing helping. The hang always happens with files > 173KB and if I use https protocol. If I revert to http protocol (still on Tomcat 9) everything works as it should (large files upload fine); switching back to https, they fail (hang). Anyone have any thoughts? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
Base64 would work. I would suggest the error log makes this explicit, so whoever looks at it knows how to deal with it and diagnose accordingly. *Manuel Dominguez Sarmiento* On 15/04/2020 15:37, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Manuel, On 4/13/20 15:13, Manuel Dominguez Sarmiento wrote: Thanks Mark. Including the request line (encoded if necessary to avoid issues with control characters) should definitely help. Yeah, I was thinking that maybe base64-encoding the request line and logging at DEBUG level would be best. I certainly don't want my log file filled-up with script kiddies all the time. - -chris On 13/04/2020 14:04, Mark Thomas wrote: On 13/04/2020 17:25, Manuel Dominguez Sarmiento wrote: Hi, we're reviewing our logs, are we are ocasionally getting the following stack traces: 09-Apr-2020 11:29:19.489 INFO [tomcat-http-81] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986 at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11In putBuffer.java:488) We understand this is a consequence of malformed requests, but we cannot seem to pinpoint the cause. It seems these are clients outside of our control (our servers are public-facing). The AccessLogValve does not log these requests, so we cannot figure out what the request line is. Is there any way logging could be improved in order to find out what is causing this? The stack trace indicates the problem is in the query string if that helps. Yes, I think we should be able to do something here. The tricky part is that as soon as an invalid character is detected we have to be a lot more careful as the payload could be malicious. I'm not sure if we'll be able to get anything into the access log but it should be possible to improve the error message and include the problematic request line in some form. You probably won't see the exact request line as we'll need to encoded things like control characters etc. I'll look at this for the May round of releases. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6XVHoACgkQHPApP6U8 pFjIMw/+MwmQMAUxdPGaiQsr87bIOsJjstnZzdePXQkBArV4C4GkxmLA6Mm4lf19 uwFhH2IhlheR3XpL/BJZeRhUGZ6sQtYzmb6fz0kAiY8VlPlUMxCTHhGp0ClL0DjY cwaL4ZZNgQsySD2aUYOzTW6pByURUfVxgkpxCu4W46q7PZ6bOgULtFnTxwoHosCz PJOliwrVUkRFAHps/vaEM87Dye2h644nDYmYDFBClBjOCmnXmGO7nj/hgMvTzA4F guHS8Tulk0ZQc87f7mYsW5XXwff+Ux7AmMzzC8q2COKiMGQv/hSQLuugn14YcKXF tRHImGWdGkYM5GIgV3FO13pb6moO6PHurh9paW6ob/8vW0HHnTcRWrBBfaqtWswx W1Z6hV83bZjiHNVrm4ZG4mlIjWmIf/QRbnl+sZgIbrmOqOsCi5P2FR4Sj0aRrplN H60Y9c0IoWbaoA/eeLA5UdvynRyFq98E4j4nmyIQyjzvDk4udOQGeiRSqmB5fxEh 8MdR+xFjc8dcNb+/mbijBRXqqCbevp5J3zCuJVafCZoYa90fgQvd7LeRjv3OnkW+ NgNzt7XcoZsb/Jp/6gdqWtRqR87acwZc0mlG/Yss/g4/Zq1ZqeCrpHuKLCiVrP1A TFA0dNhuN+ur5WAwFrjawkdVyaCQB4I7IZxtrN2X2houuvY8da4= =d/gj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AccessLogValve and IPv6 string representation (RFC 5952 section 4)
Nevermind. For some reason we had omitted this is already supported by the ipv6Canonical flag. RTFM! Manuel Dominguez Sarmiento > On 13 Apr 2020, at 20:46, Manuel Dominguez Sarmiento wrote: > > Hi, we are in the middle of a thorough review to fully support IPv6 across > our platform. It has come to our attention that Java does not fully conform > to RFC 5952 section 4 which deals with IPv6 zero compression (i.e. ::1 > instead of 0:0:0:0:0:0:0:1 for localhost). We have confirmed that Tomcat's > AccessLogValve is using the standard Java implementation. How can we > guarantee zero compression to be used in AccessLogValve? > > We are using Guava's InetAddresses.toAddrString() across our systems to deal > with this. We know we can use a custom AccessLogValve extending the standard > behaviour, but we were wondering whether there was any other solution, option > or flag around this. We've thought of using a custom request attribute to > hold the IP address, but this is not very elegant. In particular, we'd lose > the IP address if the filter we would use to set the request attribute is not > invoked for any reason. > > This is not minor, since we use access logs a lot to diagnose issues, and > cross-reference IP addresses with many other systems which are fully RFC > 5952-compliant. Having separate representations for the same IP address will > eventually lead to either trouble, misdiagnosis, missed records, etc. > > Any suggestions? > > Manuel Dominguez Sarmiento >
AccessLogValve and IPv6 string representation (RFC 5952 section 4)
Hi, we are in the middle of a thorough review to fully support IPv6 across our platform. It has come to our attention that Java does not fully conform to RFC 5952 section 4 which deals with IPv6 zero compression (i.e. ::1 instead of 0:0:0:0:0:0:0:1 for localhost). We have confirmed that Tomcat's AccessLogValve is using the standard Java implementation. How can we guarantee zero compression to be used in AccessLogValve? We are using Guava's InetAddresses.toAddrString() across our systems to deal with this. We know we can use a custom AccessLogValve extending the standard behaviour, but we were wondering whether there was any other solution, option or flag around this. We've thought of using a custom request attribute to hold the IP address, but this is not very elegant. In particular, we'd lose the IP address if the filter we would use to set the request attribute is not invoked for any reason. This is not minor, since we use access logs a lot to diagnose issues, and cross-reference IP addresses with many other systems which are fully RFC 5952-compliant. Having separate representations for the same IP address will eventually lead to either trouble, misdiagnosis, missed records, etc. Any suggestions? *Manuel Dominguez Sarmiento*
Re: java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
Thanks Mark. Including the request line (encoded if necessary to avoid issues with control characters) should definitely help. Getting through all the way to AccessLogValve would also help, but the most important bit is improving the error message. *Manuel Dominguez Sarmiento* On 13/04/2020 14:04, Mark Thomas wrote: On 13/04/2020 17:25, Manuel Dominguez Sarmiento wrote: Hi, we're reviewing our logs, are we are ocasionally getting the following stack traces: 09-Apr-2020 11:29:19.489 INFO [tomcat-http-81] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986 at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:488) We understand this is a consequence of malformed requests, but we cannot seem to pinpoint the cause. It seems these are clients outside of our control (our servers are public-facing). The AccessLogValve does not log these requests, so we cannot figure out what the request line is. Is there any way logging could be improved in order to find out what is causing this? The stack trace indicates the problem is in the query string if that helps. Yes, I think we should be able to do something here. The tricky part is that as soon as an invalid character is detected we have to be a lot more careful as the payload could be malicious. I'm not sure if we'll be able to get anything into the access log but it should be possible to improve the error message and include the problematic request line in some form. You probably won't see the exact request line as we'll need to encoded things like control characters etc. I'll look at this for the May round of releases. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
Hi, we're reviewing our logs, are we are ocasionally getting the following stack traces: 09-Apr-2020 11:29:19.489 INFO [tomcat-http-81] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986 at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:488) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) We understand this is a consequence of malformed requests, but we cannot seem to pinpoint the cause. It seems these are clients outside of our control (our servers are public-facing). The AccessLogValve does not log these requests, so we cannot figure out what the request line is. Is there any way logging could be improved in order to find out what is causing this? BTW, we're on Tomcat 9.0.33 *Manuel Dominguez Sarmiento*
Re: java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
Sorry, I was not aware of that behaviour even when changing the subject. I'll send a new, separate, unrelated message. * Manuel Dominguez Sarmiento* On 12/04/2020 16:08, Mark Thomas wrote: Please don't hijack an existing thread. Start a new message for a new topic. (Replying to an existing message and changing the header is not sufficient.) Mark On 09/04/2020 21:05, Manuel Dominguez Sarmiento wrote: Hi, we're reviewing our logs, are we are ocasionally getting the following stack traces: 09-Apr-2020 11:29:19.489 INFO [tomcat-http-81] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986 at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:488) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) We understand this is a consequence of malformed requests, but we cannot seem to pinpoint the cause. It seems these are clients outside of our control (our servers are public-facing). The AccessLogValve does not log these requests, so we cannot figure out what the request line is. Is there any way logging could be improved in order to find out what is causing this? BTW, we're on Tomcat 9.0.33 *Manuel Dominguez Sarmiento* - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
Hi, we're reviewing our logs, are we are ocasionally getting the following stack traces: 09-Apr-2020 11:29:19.489 INFO [tomcat-http-81] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986 at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:488) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) We understand this is a consequence of malformed requests, but we cannot seem to pinpoint the cause. It seems these are clients outside of our control (our servers are public-facing). The AccessLogValve does not log these requests, so we cannot figure out what the request line is. Is there any way logging could be improved in order to find out what is causing this? BTW, we're on Tomcat 9.0.33 *Manuel Dominguez Sarmiento*
Re: How to increase Memory available to Tomcat?
Using HugePages and -XX:+AlwaysPreTouch makes sure that all memory is allocated immediately. We've been using this for years to make sure our Java processes get access to all memory that would eventually be needed by the heap, metaspace, etc. Especially in order to avoid the effects of any unintended swapping, which effectively kills throughput whenever the garbage collector happens to run into memory pages that have been swapped out to disk. *Manuel Dominguez Sarmiento* On 01/04/2020 10:06, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jim, On 3/31/20 23:57, o haya wrote: I think I figured out what is going on. I think that the parameters were working all right, but the memory that is shown by the "free" command isn't corresponding to the JVM memory. I put together a small JSP that uses Runtime.getRuntime().totalMemory() and .freeMemory() and and when I access that page, that shows the JVM memory is as I configured with the Xmx/Xms. Yes, this is an error in your interpretation of your observations. Linux will lie, cheat, and steal in order to manage memory. It does that because processes lie, cheat, and steal, in order to get the resources they want. Some process says it needs a couple of gigs of RAM? Yeah, right, we'll believe it when we see it. Linux is doing its best to operate in an environment where processes often say they need resources that they don't actually use. In order to avoid prematurely running out of resources, the kernel will lie to the process and say the resources are available but not actually allocate them until the process tries to USE those resources. This is a Good Thing. free/ps/top/etc are telling you the truth. Java doesn't try to allocate the entire heap immediately, so Java thinks it can have all 4GiB of heap space and it will probably get it. But maybe not :) [1] - -chris [1] https://www.kernel.org/doc/gorman/html/understand/understand016.html On Tue, Mar 31, 2020 at 10:03 PM o haya wrote: Hi, BTW, in the catalina.out log file, I do see these lines: 01-Apr-2020 01:53:17.775 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfile.encoding=UTF-8 01-Apr-2020 01:53:17.775 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms4096m 01-Apr-2020 01:53:17.775 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx4096m 01-Apr-2020 01:53:17.775 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:NewSize=256m 01-Apr-2020 01:53:17.775 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxNewSize=256m So from that logging, it seems like Tomcat IS seeing those parameters, so why does it not seem to be using the larger amounts of memory? Thanks, Jim On Tue, Mar 31, 2020 at 9:44 PM o haya wrote: Hi, I have Tomcat installed in /apps/apache-tomcat-9.0.20/ In the /apps/apache-tomcat-9.0.20/bin/setenv.sh, I have: #!/bin/bash JAVA_HOME=/apps/jdk1.8.0_221 CATALINA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF-8 -server \ -Xms4096m -Xmx4096m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:+DisableExplicitGC \ -Djava.security.egd=file:/dev/./urandom" Then to start Tomcat, I: cd /apps/apache-tomcat-9.0.20/ bin/startup.sh Here's an example of running: [root@ip-192-168-218-224 apache-tomcat-9.0.20]# free total usedfree shared buff/cache available Mem: 32780604 91301227429012 16716 4438580 31441736 Swap: 0 0 0 [root@ip-192-168-218-224 apache-tomcat-9.0.20]# cd /apps/apache-tomcat-9.0.20/ [root@ip-192-168-218-224 apache-tomcat-9.0.20]# [root@ip-192-168-218-224 apache-tomcat-9.0.20]# bin/startup.sh Using CATALINA_BASE: /apps/apache-tomcat-9.0.20 Using CATALINA_HOME: /apps/apache-tomcat-9.0.20 Using CATALINA_TMPDIR: /apps/apache-tomcat-9.0.20/temp Using JRE_HOME: /apps/jdk1.8.0_221 Using CLASSPATH: /apps/apache-tomcat-9.0.20/bin/bootstrap.jar:/apps/apache-tomcat-9.0 .20/bin/tomcat-juli.jar Tomcat started. [root@ip-192-168-218-224 apache-tomcat-9.0.20]# free total usedfree shared buff/cache available Mem: 32780604 134328826998784 16716 4438532 31011556 You are right that it appears like it is not using the CATALINA_OPTS to set the memory, but I don't know why ??? Thanks, Jim On Tue, Mar 31, 2020 at 12:52 PM Mark Thomas wrote: On 31/03/2020 17:42, o haya wrote: Thanks for the infos. I was thinking the same thing you were thinking, that it might be 32-bit JVM, but when I run "java -version" I am getting: java -version java version "1.8.0_221" Java(TM) SE Runtime Environment (build 1.8.0_221-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode) Isn't that confirming that this is a 64-bit JVM? ALSO FYI, I did test where I also added "-d64" parameter to th
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Thanks, so we would be able to log this with AccessLogValve using the
following patterns, right?
%{org.apache.coyote.connectionID}r
%{org.apache.coyote.streamID}r
*Manuel Dominguez Sarmiento*
On 31/03/2020 10:28, Mark Thomas wrote:
On 29/03/2020 16:16, Mark Thomas wrote:
On 29/03/2020 15:58, Manuel Dominguez Sarmiento wrote:
Do the streams with errors all share the same connection ID? I suspect
an error on one stream is triggering the closure of the connection and
what you are seeing is the fallout of closing the connection.
That makes sense and is kind of what we're suspecting. How can we check
the connection/stream id? Can this be logged by AccessLogValve?
Not easily at the moment. That looks like something we should fix. I'll
see what I can do for 9.0.34.
From 8.5.54, 9.0.34 and 10.0.0-M4 the following request attributes will
be available for HTTP/2
org.apache.coyote.connectionID
org.apache.coyote.streamID
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Do the streams with errors all share the same connection ID? I suspect an error on one stream is triggering the closure of the connection and what you are seeing is the fallout of closing the connection. That makes sense and is kind of what we're suspecting. How can we check the connection/stream id? Can this be logged by AccessLogValve? *Manuel Dominguez Sarmiento *
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Mark, we're now on the latest 9.0.33 release and we still see this
issue intermittently in our logs. Only on HTTP/2 secure requests.
Please see the attached access logs (these represent all the cases for
one whole day in one single high-volume server).
Some of the following request fields are NULL (or -1) in these examples:
- remoteAddr
- remotePort
- serverPort
- requestURI
- User-Agent
Some requests are missing some of the fields, some of the requests are
missing others. What is particularly interesting is that the errors are
clustered around particular timestamps, pointing to some likely issue
regarding object sharing across several requests.
Please note that this is not just an issue at the AccessLogValve level.
These fields contain invalid data while the request is being processed,
so that is causing unexpected exceptions in our production code. The
cases are few and isolated, but still this should be looked into.
Any thoughts?
*Manuel Dominguez Sarmiento*
On 05/02/2020 14:12, Manuel Dominguez Sarmiento wrote:
Our filter is not doing anything fancy (and it has always worked
correctly before we ran into this bug). In pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr();
boolean isProxy = isProxy(ip);
if (isProxy) {
String unwrappedIP = unwrapXForwardedFor(request);
chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response);
} else {
chain.doFilter(request, response);
}
}
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating to the
actual request, while unwrapXForwardedFor() does what the name
suggests, which is processing X-Forwarded-For to obtain the
originating IP before it hit the detected proxy.
*Manuel Dominguez Sarmiento*
On 05/02/2020 10:28, Mark Thomas wrote:
On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote:
We are getting the NPEs in a top-of-the-chain servlet filter which
decorates HttpServletRequest.getRemoteAddr() before actual servlet
processing. Only on HTTP/2 and in a very small number of cases. Perhaps
we should test 9.0.31 and see what happens. When is this new version due
for release?
I'm just working through back-porting some changes and then I'll be
starting the release process. It 9.0.31 should be available towards the
beginning of next week.
Can you expand on what your filter is doing? When is the call made to
HttpServletRequest.getRemoteAddr() on the original request?
Mark
LOOKING FOR ALL ISSUE INSTANCES:
[root@optimus ~]# cat /home/wap/logs/access.2020-03-27.log | grep "^-"
- -1 443 [27/Mar/2020:07:53:12 -0300] "GET /us/en/country.do?method=list
HTTP/2.0" 400 762 "-" "Mozilla/5.0 (Linux; Android 6.0; vivo 1609)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36"
- -1 443 [27/Mar/2020:10:48:12 -0300] "GET
/pe/es/subscriptionPlanDetail.do?id=4483=false=2181=46045=true=419634618870==ojo.pe=d=EAIaIQobChMIif6cyOW66AIVKAa5Bh3eRgI6EAEYASAAEgJuRPD_BwE
HTTP/2.0" 400 637 "https://ojo.pe/; "Mozilla/5.0 (Linux; Android 9; LM-X520
Build/PKQ1.190223.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0
Chrome/80.0.3987.119 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/260.0.0.42.118;]"
- -1 443 [27/Mar/2020:14:39:36 -0300] "GET
/cl/es/subscriptionPlanDetail.do?id=4120=false=2131=17450=false=45663=true=380011499904==mobileapp%3A%3A2-com.appstar.callrecorder=EAIaIQobChMI-aa0_8uN6AIVZga5Bh1UBwQ5EAEYASAAEgKAxPD_BwE_cl_smd_ok=32320413=32320413=705b26c82e98b8401b74a463a68180d6=1584044911681=CELLULAR=EFFECTIVE_4G=true
HTTP/2.0" 400 637
"https://wap.renxo.com/cl/es/subscriptionPlanDetail.do?id=4120=false=2131=17450=false=45663=true=380011499904==mobileapp%3A%3A2-com.appstar.callrecorder=EAIaIQobChMI-aa0_8uN6AIVZga5Bh1UBwQ5EAEYASAAEgKAxPD_BwE_cl_smd_ok=32320413=32320413=705b26c82e98b8401b74a463a68180d6=1584044911681=CELLULAR=EFFECTIVE_4G;
"Mozilla/5.0 (Linux; Android 8.1.0; SAMSUNG SM-J710MN Build/M1AJQ)
AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/9.4 Chrome/67.0.3396.87
Mobile Safari/537.36"
- -1 443 [27/Mar/2020:17:18:55 -0300] "GET
/ar/es/subscriptionPlanDetail.do?id=4328=16242=2403=48008=true=409370554249=%2Farts%20%26%20entertainment=cuttsite.website=d=EAIaIQobChMI98zj67y76AIVT4p3Ch3riAVXEAEYASAAEgLGavD_BwE
HTTP/2.0" 400 637
"https://googleads.g.doubleclick.net/pagead/ads?client=ca-pub-2929318494915332=html=250=4353673900=3696358127=3108995893=300=1585340298=1=2.2.0.0.2.2.0.0=300x250=https%3A%2F%2Fcuttsite.website%2Fpica-3-limones-con-sal-encima-ponlo-cerca-de-tu-cama-y-veras-lo-que-pasa-en-los-proximos-15-minutos%2F=0=1=1585340297517=36=2610=526=r20200324=r20190131=9=aa=1=ID%3D7489a82afd59cfe7%3AT%3D1580676408%3AS%3DALNI_MbJWniZS34uo5D_IKexygyl9kHRXg=1_fmts=360x300=3721745222895=20=1_vid=662555076.1580676587_sid=1
Re: Uploads breaking post upgrade to 9.0.31
Great, I just saw that :-) On 17/03/2020 11:24, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Manuel. On 3/17/20 09:25, Manuel Dominguez Sarmiento wrote: Hi Mark, when is 9.0.32 expected to be released? We've seen this issue reported by several users, even if we haven't run into this particular case directly (yet) 9.0.33 was announced about 20 minutes ago :) - -chris On 17/03/2020 09:51, Mark Thomas wrote: https://bz.apache.org/bugzilla/show_bug.cgi?id=64202 Mark On 17/03/2020 11:46, Srijith Kochunni wrote: Hi All, This is to seek help on a strange issue that we are observing. We recently did a minor upgrade of Tomcat from 9.0.30 to 9.0.31, in our application, in order to address vulnerability in AJP connector. Ever since then we have started seeing upload failures with our upload servlet when processing large files. Small files do get uploaded, but when we upload large files and we do Multipart file upload, we are randomly and yet consistently seeing that we get the following exception. [org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUpload Base.java:351) Caused by: org.apache.commons.fileupload.MultipartStream$MalformedStreamExcepti on: Stream ended unexpectedly at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAv ailable(MultipartStream.java:1005) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(M ultipartStream.java:903) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:100) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:70) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUpload Base.java:347) It appears that the connection is getting reset in the middle of the upload, but the client is very much up and we get PR_CONNECT_RESET_ERROR on the browser. My code on the server side is as simple as DiskFileItemFactory factory = new DiskFileItemFactory(); ServletFileUpload fileUpload = new ServletFileUpload(factory); List fileItems = fileUpload.parseRequest(originalRequest); We would like to know if anyone else has observed this and if there is any way we can debug this further. When I try to attach and debug, the upload however seems to go through fine and is only failing when I am not attached to the process. Any help / suggestions would be much appreciated. Thanks, Srijith. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5w3bkACgkQHPApP6U8 pFg/gg/+MHNKYcFiWA3njQuNxqY2DRumdXryFIep9Ezi6L7KpLAwfGSpi+BMZLew 53d+JjWPhjLebjB2zEQAXdXvl9fHtHWDJoH4QKXYcjm7Lljj1ZpsGNR99EPWV1hX dS0aqPo0bfq7cjlg3Euh1vxC+BLccIJOvpC1l4L/UhTkCfDP8O5Yzy8KXkZVl9q3 AFSIHOjC09/1Z51QHHBrOsbuRkN69/Ouuks9pTGA4A53xjN0jBYyiBa8NaOoxt5U M4H3ipfdDJf41lwPbhBZ51dip0EAh6frI1tWkDKFkJsyms3Byj6sE4sLLw+ViSLy H0FvbAw75nFBeZlO7Fl0IKKgFxtHaJmMhfBf2sXzkorEv0SQVE/c/5CO4ry+to0O +9HdtkXXRVfaeCfCdltyvMAWOPDuFGF1Y2MKwFeHT1c/nDcMuzhEUP6QS8SKAJc+ uEsPe4PiJ4441MOF+E9Nj4SpKgfdtnL5M3r36N/Yad23eQGRzhzaB5c6uoPd6HAY TCcdxf1BXaX+RxjQFbwG4xwmGLrmYiH99tZJ63xm47KbVONdvKCJX5lovcOYiheJ 3PgqNSLojcjMwreyBFVe6oK/CKKolp4i5sEJsfc9GpegBj73aRuMHccaS5RTAn+B c22P1jzpHbCPkrl5M7dIe00keMAixF0TZ/wWOqp2yMdL8Rx2nUc= =K4Sn -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Uploads breaking post upgrade to 9.0.31
Hi Mark, when is 9.0.32 expected to be released? We've seen this issue reported by several users, even if we haven't run into this particular case directly (yet) On 17/03/2020 09:51, Mark Thomas wrote: https://bz.apache.org/bugzilla/show_bug.cgi?id=64202 Mark On 17/03/2020 11:46, Srijith Kochunni wrote: Hi All, This is to seek help on a strange issue that we are observing. We recently did a minor upgrade of Tomcat from 9.0.30 to 9.0.31, in our application, in order to address vulnerability in AJP connector. Ever since then we have started seeing upload failures with our upload servlet when processing large files. Small files do get uploaded, but when we upload large files and we do Multipart file upload, we are randomly and yet consistently seeing that we get the following exception. [org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:351) Caused by: org.apache.commons.fileupload.MultipartStream$MalformedStreamException: Stream ended unexpectedly at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:1005) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:903) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:100) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:70) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:347) It appears that the connection is getting reset in the middle of the upload, but the client is very much up and we get PR_CONNECT_RESET_ERROR on the browser. My code on the server side is as simple as DiskFileItemFactory factory = new DiskFileItemFactory(); ServletFileUpload fileUpload = new ServletFileUpload(factory); List fileItems = fileUpload.parseRequest(originalRequest); We would like to know if anyone else has observed this and if there is any way we can debug this further. When I try to attach and debug, the upload however seems to go through fine and is only failing when I am not attached to the process. Any help / suggestions would be much appreciated. Thanks, Srijith. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Christopher, please see below: 1) The Tomcat valves operate on all webapps. We only need/require this for one particular webapp without affecting the others. Not true; see Konstantin's response. Yes, I realized per-context valve configuration was possible after I sent my original email. 2) The code has been simplified for illustration purposes. Besides X-Forwarded-For, we detect and work around many other custom external mobile proxies which do not use X-Forwarded-For and require custom Geolocation code to detect the ISP and connection type (Google Compression Proxy, Nokia OVI, Novarra, Lotus Flare, Opera Mini, Opera Max, Samsung Max, etc.) - this kind of customization is not possible without custom code. Interesting. Is this something you think would be widely useful Not sure how useful it would be to other users. We are a fairly specialized shop running platforms for mobile carriers, so mobile carrier detection, MSISDN header enrichment, detecting whether users are on a real 3G/4G network or through a modem, hotspot or proxy, etc. are a major concern for us, but I don't believe most users would care about this. and/or would be willing to share with the community? We'd be willing to share our code, however see my comments above. If it's a fast-moving target (e.g. new public proxies are popping-up all the time, or existing proxies keep changing their configuration requirements) then maybe it's not a great fit for a stable product like Tomcat. Most of our proxy detection logic works by checking geolocation data from trusted, paid sources (MaxMind and Digital Element/NetAcuity) so even if the logic is simpe, it requires fresh, updated data from a trusted third-party to be of any use. On the other hand, some of the proxy detection logic instead depends on proprietary header analysis (for instance, Opera Mini inserts many X-OperaMini- prefixed headers). On the other hand, if it could be configured relatively easily (like with a "proxy definitions" file or something), then it could still be very valuable even with a simple or default configuration which only supports some very large proxies (e.g. Akamai, CloudFront, CloudFlare, etc.). This is only useful for small, well-known partner proxies that are supposed to notify us if their IP addresses change. But major proxies change their IP ranges all the time. So, again, this would not really be useful without the third-party geolocation data. This filter is not meant for detecting internal proxies within our control (such as Apache front ends or load balancers), but rather public proxies which are "transparently" (not really) used via some mobile devices and services. Does it matter whether these are "internal" versus "external" proxies? The only real difference is the IP-range of the proxy, right? No, that's not the only difference. Some proxies introduce non-standard behaviour, even if they advertise themselves as "trasparent": - Novarra proxy transcodes HTML for small screens (this proxy is obsolete, but is a prime example of non-transparency) - Opera Mini does all sorts of fancy things to compress responses: it coalesces HTML, images, and JavaScript payloads to minimize the number of streams; modifies HTML to fit devices with small screens and/or low memory/CPU resources, etc. - Chrome Compression Proxy (aka "Data Saver mode") is fairly transparent, but there are many caveats. In particular, MSISDN header enrichment breaks, which is a major issue for services we manage for mobile carriers. See https://developer.chrome.com/multidevice/data-compression-for-isps for more details. - etc. It doesn't matter whether you control the proxy or it's an external service: you still have to secure and validate the connection in the same ways, and take the same action(s) on the server-end where you trust the information being presented. See comments above. *Manuel Dominguez Sarmiento*
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Konstantin, please see below: You can configure a Valve for a specific web application by placing it into Context configuration for that specific web application (usually that is the "/META-INF/context.xml" configuration file). [1] [1] http://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Defining_a_context You are correct. I realized this after my original response. We actually already use per-context AccessLogValves. 2. If I understand correctly, the null value from request.getRemoteAddr() means that the client connection has already been closed. Tomcat cannot do much at that point, unless the information has not already been requested (and thus cached) when the connection was still alive. (The recent changes to the AccessLogValve are just that: to request the value earlier.) It is useless to process a request if the connection has already been closed. We request getRemoteAddr() several times during servlet processing (for Geolocation, proxy and carrier detection among other purposes), as well as in top-of-the-chain servlet filters. If the connection has been closed, it must be because of some Tomcat issue. This never happened with earlier Tomcar versions (we started this project many years ago with 6.x). Plus, the issue is only present in a small number of cases on HTTP/2, but not regular HTTPS nor plain HTTP. Why do you say that the null value is an invalid one? I do not see such words in the specification. I disagree. There cannot be a TCP/IP, HTTP, HTTPS, HTTP/2 or HTTP/3-QUIC connection without an originating IP, so it makes no sense to return NULL under any circumstances. Furthermore, the spec Javadoc for getRemoteAddr() states "Returns: a String containing the IP address of the client that sent the request" Contrast this to getRemoteUser() for instance, which states: "Returns: a String specifying the login of the user making this request, *or null if the user login is not known*" If NULL were a valid return value, it would be explicitly mentioned. 3. Just as a note (I would not recommend it for your specific use case) One known way to detect a closed connection is to trigger parameter parsing and look whether an error flag (implemented as an attribute of a Request) was set by it. See the implementation of org.apache.catalina.filters.FailedRequestFilter for an example. As a workaround, since we only have a very small number of requests that trigger this, we have implemented a filter that discards those requests and returns HTTP 400 "Bad Request", until the issue is resolved. 4. Do you run with the following configuration setting turned on? org.apache.catalina.connector.RECYCLE_FACADES=true We do not use this option, so we must be running with the default="false" See https://cwiki.apache.org/confluence/x/yColBg#TroubleshootingandDiagnostics-TroubleshootingunexpectedResponsestateproblems Thanks, I'll read through this. *Manuel Dominguez Sarmiento*
Re: [OT] HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Yes, there are two reasons:
1) The Tomcat valves operate on all webapps. We only need/require this
for one particular webapp without affecting the others.
2) The code has been simplified for illustration purposes. Besides
X-Forwarded-For, we detect and work around many other custom external
mobile proxies which do not use X-Forwarded-For and require custom
Geolocation code to detect the ISP and connection type (Google
Compression Proxy, Nokia OVI, Novarra, Lotus Flare, Opera Mini, Opera
Max, Samsung Max, etc.) - this kind of customization is not possible
without custom code. This filter is not meant for detecting internal
proxies within our control (such as Apache front ends or load
balancers), but rather public proxies which are "transparently" (not
really) used via some mobile devices and services.
*
Manuel Dominguez Sarmiento*
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Manuel,
On 2/5/20 12:12 PM, Manuel Dominguez Sarmiento wrote:
Our filter is not doing anything fancy (and it has always worked
correctly before we ran into this bug). In pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr(); boolean isProxy =
isProxy(ip); if (isProxy) { String unwrappedIP =
unwrapXForwardedFor(request); chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request, unwrappedIP),
response); } else { chain.doFilter(request, response); } }
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating to the
actual request, while unwrapXForwardedFor() does what the name
suggests, which is processing X-Forwarded-For to obtain the
originating IP before it hit the detected proxy.
Any reason not to use the valves Tomcat provides to do pretty much
this exact thing?
https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_Val
ve
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl47AfoACgkQHPApP6U8
pFgywBAAxq4KZaxhO/X0EaoA3G3TDKURbDSrA2Hbke0QVgP3tX6QX6MDMEkbqTuv
x4S1xgXeNgWNIE9Dy62t4ciC6VZ21v6Rl4JMTOmtUO0ES+OBNTSYXstA1ZywBoMD
bRXHNjjdPtxAhhqQblfpKter+vn/k/PZ+45GgZSLNMrrJR+1idAafWs7YwpCHxfL
zh+C5C3uHu+bznCiOF/NQQ5d3/0mHyOj5nSpYx2ImSH+rgj0/Ch5w43L4chD1Lqj
aw2sqqC2z4Fzp3R8uNX/uMRV4flck9KILrfOe00BrMGVfPVuDDGa2J4NhSTs0oNx
Pda5j19QAP6wh6vIKH5KVM8fUWcb5KthcT34cxKttFl3gL0EedE40y97PHKxihJ5
BaT9M3MD0Lsh/mtgrSGcxHU1G0Vs5hHw9e9Zm0XqgQeLsuIALnSHcFyQcGBY05sH
9jljk/l+iMIp9xi3fXyCgYHRpCvbwOuT92V4pMYmwLCzvpfIrVspTi+7AxzbipBV
7tTvH4om0PSchj+Gj3ayOkQZsAyDZbJXbV4izxN3p1c8AJT5ZUmJnYqpKNtfAJpW
w/3OxIXoTUq3bbpb6AHrJe36+QU6+/C2WNqLBcevElr6T1oObKfzy1CjoUHxUQCm
wWxrSFac6atc56mVuPYAjs1DZ0sKpvts/ih6AwDF8+0jBzeJX+g=
=5GpE
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Our filter is not doing anything fancy (and it has always worked
correctly before we ran into this bug). In pseudo-code:
public doFilter(request, response) {
String ip = request.getRemoteAddr();
boolean isProxy = isProxy(ip);
if (isProxy) {
String unwrappedIP = unwrapXForwardedFor(request);
chain.doFilter(new
MobileProxyHidingServletRequestWrapper(request, unwrappedIP), response);
} else {
chain.doFilter(request, response);
}
}
All that MobileProxyHidingServletRequestWrapper is override
getRemoteAddr() returning unwrappedIP instead of delegating to the
actual request, while unwrapXForwardedFor() does what the name suggests,
which is processing X-Forwarded-For to obtain the originating IP before
it hit the detected proxy.
*Manuel Dominguez Sarmiento*
On 05/02/2020 10:28, Mark Thomas wrote:
On 04/02/2020 22:27, Manuel Dominguez Sarmiento wrote:
We are getting the NPEs in a top-of-the-chain servlet filter which
decorates HttpServletRequest.getRemoteAddr() before actual servlet
processing. Only on HTTP/2 and in a very small number of cases. Perhaps
we should test 9.0.31 and see what happens. When is this new version due
for release?
I'm just working through back-porting some changes and then I'll be
starting the release process. It 9.0.31 should be available towards the
beginning of next week.
Can you expand on what your filter is doing? When is the call made to
HttpServletRequest.getRemoteAddr() on the original request?
Mark
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
We are getting the NPEs in a top-of-the-chain servlet filter which decorates HttpServletRequest.getRemoteAddr() before actual servlet processing. Only on HTTP/2 and in a very small number of cases. Perhaps we should test 9.0.31 and see what happens. When is this new version due for release? *- **Manuel Dominguez Sarmiento* Thanks Mark. I just wanted to clarify that the issue is not only present when the request arrives at AccessLogValve, but while the request is being serviced as well. We noticed this bug because we were getting random NullPointerExceptions when trying to do anything with the result from HttpServletRequest.getRemoteAddr() in our servlets. Should the fix solve this as well? It depends. The only way I can see this happening is if the application swallows an IOException and continues. It probably won't fix that and I don't think we'd consider an NPE an error in those circumstances. Mark
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Thanks Mark. I just wanted to clarify that the issue is not only present when the request arrives at AccessLogValve, but while the request is being serviced as well. We noticed this bug because we were getting random NullPointerExceptions when trying to do anything with the result from HttpServletRequest.getRemoteAddr() in our servlets. Should the fix solve this as well? - Manuel Dominguez Sarmiento On 03/02/2020 19:24, Mark Thomas wrote: I haven't fixed this but I can reproduce it easily with the h2spec test suite. As I have a reproducible test case I'm hopeful I'll be able to find a fix fairly quickly. I've just committed a fix. It is in: - master for 10.0.0.0-M1 onwards - 9.0.x for 9.0.31 onwards - 8.5.x for 8.5.51 onwards 7.0.x was not affected. The issue was that by the time the AccessLogValve tried to obtain the remote host/addr/port the connection had been closed and the information was not available. This was a result of the change below which removed a potential path for a request to see incorrect remote client information. This entry in the changelog: Decouple the socket wrapper, which is not recycled, from the NIOx channel after close, and replace it with a dummy static object. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi Mark, thanks for your feedback. Please see below: On 23/01/2020 13:40, Manuel Dominguez Sarmiento wrote:> Hi, we started noticing that HttpServletRequest.getRemoteAddr() was sometimes returning NULL (which is invalid according to the servlet spec), about 20-50 times per day (we have high-load servers which routinely handle over 100 requests per second). This only happens on secure HTTP/2 requests. We noticed this while running the latest 9.0.30, but looking at our logs for the past few months, this was also happening on 9.0.21 as well as 9.0.22 which are the previous versions we had deployed. This issue was not present in 9.0.17, which was the version we were using before 9.0.21. Before anyone cries out "there is a bug in your code", Tomcat itself reports a NULL remoteAddr via AccessLogValve in these cases. It is also interesting that the remotePort is reported as "-1". We looked through the changelog, but we can't figure out what change might have triggered this (presumably some change between 9.0.17 and 9.0.21). It looks like some kind of async race condition during the HTTP/2 upgrade, but this is only an educated guess. I'll take a look. You mention async. Are these requests using the Servlet async API? No, we're not using the async API in any capacity. Just the regular servlet API: doGet() / doPost() / etc. Perhaps my usage of "async" was confusing (as that can refer to several different things in Tomcat). I just meant that it feels like a race condition between different threads. Our server.xml is below (confidential data has been modified). We are using the NIO connector, Tomcat Native + APR libraries (but NOT the APR connector). The useAsyncIO flag is disabled because of server lockups we've experienced with this flag enabled on our production servers. Any ideas? Nothing comes to mind immediately. Mark Please let me know if there's any diagnostic info that might help with this and we'll produce it. Thanks again, - Manuel Dominguez Sarmiento
HttpServletRequest.getRemoteAddr() sometimes returns NULL on Tomcat 9.0.30 and HTTP/2 secure requests
Hi, we started noticing that HttpServletRequest.getRemoteAddr() was
sometimes returning NULL (which is invalid according to the servlet
spec), about 20-50 times per day (we have high-load servers which
routinely handle over 100 requests per second). This only happens on
secure HTTP/2 requests.
We noticed this while running the latest 9.0.30, but looking at our logs
for the past few months, this was also happening on 9.0.21 as well as
9.0.22 which are the previous versions we had deployed. This issue was
not present in 9.0.17, which was the version we were using before 9.0.21.
Before anyone cries out "there is a bug in your code", Tomcat itself
reports a NULL remoteAddr via AccessLogValve in these cases. It is also
interesting that the remotePort is reported as "-1".
We looked through the changelog, but we can't figure out what change
might have triggered this (presumably some change between 9.0.17 and
9.0.21). It looks like some kind of async race condition during the
HTTP/2 upgrade, but this is only an educated guess.
Our server.xml is below (confidential data has been modified). We are
using the NIO connector, Tomcat Native + APR libraries (but NOT the APR
connector). The useAsyncIO flag is disabled because of server lockups
we've experienced with this flag enabled on our production servers. Any
ideas?
- Manuel Dominguez Sarmiento
Http11NioProtocol deadlocks with useAsyncIO="true" and HTTPS/2.0
Hi, we had been running Tomcat 9.0.17 for quite some time on our high-load production servers, using the attached server.xml configuration. Upon upgrading to 9.0.21 we started experiencing many random deadlocks. We run performance advertising campaigns, and our conversion rates dropped to below half of what they usually are, which was an obvious consequence of our servers randomly locking up. Plus, it was very easy to reproduce the deadlocks, which seemed to "magically unlock" when opening a second tab/window and opening the same URL that was locked on the other window/tab. Doing this unlocked both windows/tabs at once, immediately. We found this was only happening on HTTPS, but NOT on HTTP. Furthermore, we found this was only happening when the browser negotiated an upgrade to HTTPS/2.0 Once we found this, we temporarily removed the className="org.apache.coyote.http2.Http2Protocol" /> configuration, and all was back to normal. However, we need HTTP/2, so we continued to look for a proper solution. Looking at the Tomcat changelog, we found there have been many changes since 9.0.17 related to useAsyncIO and HTTP/2. One particular change for 9.0.22 caught our attention: /"Remove a source of potential deadlocks when using HTTP/2 when the Connector is configured with useAsyncIO as true. (markt)" /We also found the following discussion thread, which describes issues similar to what we were experiencing:/ /http://mail-archives.apache.org/mod_mbox/tomcat-dev/201906.mbox/%3c20190606204631.bab6c8a...@gitbox.apache.org%3e/ / So we upgraded to 9.0.22 thinking that the deadlock would be gone. But alas, it was not. The deadlocks remained. We found that 9.0.20 changed the default for useAsyncIO from "false" to "true". So we changed useAsyncIO back to what it was when we were running 9.0.17 (false) and all is back to normal on 9.0.22 So the conclusion is: there are still deadlock bugs on the NIO connector with useAsyncIO="true" and upgrades to HTTP/2.0 Besides fixing them, we believe that the useAsyncIO default should be reverted to "false". We could find no Java deadlocked threads at all by inspecting jconsole (not with the automatic "find deadlocks" functionality, nor by inspection of a thread dump). We performed several thread dumps WHILE the deadlock was clearly visible on screen (this was very easily reproduceable). The deadlock is definitely there though and goes away as soon as we turn off "useAsyncIO". Since we could not find Java-level deadlocks, we believe the problem probably lies in the interaction with native code. We are using org.apache.catalina.core.AprLifecycleListener as well as Tomcat Native 1.2.23 on Linux. We could not find any pointers in the Tomcat Native changelog dealing with similar issues. Any ideas? Thanks, -- Manuel Dominguez Sarmiento - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
