Re: Firefox triggers HTTP2 overhead protection - known issue?
Leon, On 5/6/21 12:35, Leon Atherton wrote: On 06/05/2021 17:13, Leon Atherton wrote: On 06/05/2021 16:06, Christopher Schultz wrote: On 5/6/21 09:36, Mark Thomas wrote: On 06/05/2021 13:33, Christopher Schultz wrote: Leon, On 5/6/21 06:25, Leon Atherton wrote: We are seeing that Firefox triggers the HTTP2 overhead protection with multipart file uploads. About 1MB is uploaded before overhead protection is triggered. I believe a few weeks ago Chrome was triggering this too, but it looks like a recent update may have resolved it. This is on Tomcat 9.0.45 (though it applies to earlier versions too) using the org.apache.coyote.http11.Http11NioProtocol connector with upgrade to org.apache.coyote.http2.Http2Protocol. With http2 debug logging enabled, there are occasional Payload size [1] interspersed between larger payloads, though it appears to be a couple of back-to-back Payload size [1]'s which may be triggering the org.apache.coyote.http2.ConnectionException: Connection [0], Too much overhead so the connection will be closed. I've found earlier Tomcat conversations which seem to indicate it's a client issue. I've searched through the Firefox bug tracker but have not found anything that looks like this (feels like looking for a needle in a haystack). Is this a known issue in Tomcat or Firefox? It can be resolved by disabling the protection or potentially just tweaking the config values, but if it's a known Firefox issue I'd like to know if something has been raised on their bug tracker I can follow? Nothing in the tracker that I know of off-hand, but there is this informative answer on SO by markt which may help you a little: https://stackoverflow.com/a/61454503/276232 Using the information in that answer may allow you to reconfigure your server to allow Firefox's behavior. It's probably worth us taking some time to adapt markt's SO answer there into a whole section on "Protocol Abuse and Protection Features" in the HTTP/2 configuration guide. There is an open issue for Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=1000809 ACK but there hasn't been much movement. I'll add something to my TODO list to see what can be done to avoid false positives like this as it looks like short sequences of small packets can occur depending on how the client buffers data. Maybe a tweak to the default settings can improve things for "the status quo" for current web browser behavior. Thanks guys. I'm currently unable to reproduce the problem in Chrome (not seeing any 1 byte payloads). I thought it reproduced a few weeks ago, but perhaps I'm misremembering. Firefox's market share isn't what it was, but it would be good if this could work out of the box - either by tweaking the Tomcat code or default config to prevent the false-positive or opening a ticket so the Firefox team know what to fix. Happy to help with the latter, though I'd need some assistance to explain the problem and expected solution. I could bore you to tears with the ins and outs of the PDF spec, but HTTP2 is not one of my specialist subjects. Leon Just to confirm that I have been able to reproduce this on an older build of Chromium (88.0.4324.0). Sounds like more recent builds of Chrome have addressed this issue, or have at least made the h2 component more well-behaved. Seems like it's the same issue as the current version of Firefox - consequtive 1 byte payloads triggering the overhead protection. Firefox may be working on tweaking this stuff. I would file a bug against ff or comment on any existing related ones you may be able to find. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
On 06/05/2021 17:13, Leon Atherton wrote: On 06/05/2021 16:06, Christopher Schultz wrote: On 5/6/21 09:36, Mark Thomas wrote: On 06/05/2021 13:33, Christopher Schultz wrote: Leon, On 5/6/21 06:25, Leon Atherton wrote: We are seeing that Firefox triggers the HTTP2 overhead protection with multipart file uploads. About 1MB is uploaded before overhead protection is triggered. I believe a few weeks ago Chrome was triggering this too, but it looks like a recent update may have resolved it. This is on Tomcat 9.0.45 (though it applies to earlier versions too) using the org.apache.coyote.http11.Http11NioProtocol connector with upgrade to org.apache.coyote.http2.Http2Protocol. With http2 debug logging enabled, there are occasional Payload size [1] interspersed between larger payloads, though it appears to be a couple of back-to-back Payload size [1]'s which may be triggering the org.apache.coyote.http2.ConnectionException: Connection [0], Too much overhead so the connection will be closed. I've found earlier Tomcat conversations which seem to indicate it's a client issue. I've searched through the Firefox bug tracker but have not found anything that looks like this (feels like looking for a needle in a haystack). Is this a known issue in Tomcat or Firefox? It can be resolved by disabling the protection or potentially just tweaking the config values, but if it's a known Firefox issue I'd like to know if something has been raised on their bug tracker I can follow? Nothing in the tracker that I know of off-hand, but there is this informative answer on SO by markt which may help you a little: https://stackoverflow.com/a/61454503/276232 Using the information in that answer may allow you to reconfigure your server to allow Firefox's behavior. It's probably worth us taking some time to adapt markt's SO answer there into a whole section on "Protocol Abuse and Protection Features" in the HTTP/2 configuration guide. There is an open issue for Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=1000809 ACK but there hasn't been much movement. I'll add something to my TODO list to see what can be done to avoid false positives like this as it looks like short sequences of small packets can occur depending on how the client buffers data. Maybe a tweak to the default settings can improve things for "the status quo" for current web browser behavior. Thanks guys. I'm currently unable to reproduce the problem in Chrome (not seeing any 1 byte payloads). I thought it reproduced a few weeks ago, but perhaps I'm misremembering. Firefox's market share isn't what it was, but it would be good if this could work out of the box - either by tweaking the Tomcat code or default config to prevent the false-positive or opening a ticket so the Firefox team know what to fix. Happy to help with the latter, though I'd need some assistance to explain the problem and expected solution. I could bore you to tears with the ins and outs of the PDF spec, but HTTP2 is not one of my specialist subjects. Leon Just to confirm that I have been able to reproduce this on an older build of Chromium (88.0.4324.0). Seems like it's the same issue as the current version of Firefox - consequtive 1 byte payloads triggering the overhead protection. Glad I'm not going crazy. Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
On 06/05/2021 16:06, Christopher Schultz wrote: On 5/6/21 09:36, Mark Thomas wrote: On 06/05/2021 13:33, Christopher Schultz wrote: Leon, On 5/6/21 06:25, Leon Atherton wrote: We are seeing that Firefox triggers the HTTP2 overhead protection with multipart file uploads. About 1MB is uploaded before overhead protection is triggered. I believe a few weeks ago Chrome was triggering this too, but it looks like a recent update may have resolved it. This is on Tomcat 9.0.45 (though it applies to earlier versions too) using the org.apache.coyote.http11.Http11NioProtocol connector with upgrade to org.apache.coyote.http2.Http2Protocol. With http2 debug logging enabled, there are occasional Payload size [1] interspersed between larger payloads, though it appears to be a couple of back-to-back Payload size [1]'s which may be triggering the org.apache.coyote.http2.ConnectionException: Connection [0], Too much overhead so the connection will be closed. I've found earlier Tomcat conversations which seem to indicate it's a client issue. I've searched through the Firefox bug tracker but have not found anything that looks like this (feels like looking for a needle in a haystack). Is this a known issue in Tomcat or Firefox? It can be resolved by disabling the protection or potentially just tweaking the config values, but if it's a known Firefox issue I'd like to know if something has been raised on their bug tracker I can follow? Nothing in the tracker that I know of off-hand, but there is this informative answer on SO by markt which may help you a little: https://stackoverflow.com/a/61454503/276232 Using the information in that answer may allow you to reconfigure your server to allow Firefox's behavior. It's probably worth us taking some time to adapt markt's SO answer there into a whole section on "Protocol Abuse and Protection Features" in the HTTP/2 configuration guide. There is an open issue for Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=1000809 ACK but there hasn't been much movement. I'll add something to my TODO list to see what can be done to avoid false positives like this as it looks like short sequences of small packets can occur depending on how the client buffers data. Maybe a tweak to the default settings can improve things for "the status quo" for current web browser behavior. Thanks guys. I'm currently unable to reproduce the problem in Chrome (not seeing any 1 byte payloads). I thought it reproduced a few weeks ago, but perhaps I'm misremembering. Firefox's market share isn't what it was, but it would be good if this could work out of the box - either by tweaking the Tomcat code or default config to prevent the false-positive or opening a ticket so the Firefox team know what to fix. Happy to help with the latter, though I'd need some assistance to explain the problem and expected solution. I could bore you to tears with the ins and outs of the PDF spec, but HTTP2 is not one of my specialist subjects. Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Mark, On 5/6/21 09:36, Mark Thomas wrote: On 06/05/2021 13:33, Christopher Schultz wrote: Leon, On 5/6/21 06:25, Leon Atherton wrote: We are seeing that Firefox triggers the HTTP2 overhead protection with multipart file uploads. About 1MB is uploaded before overhead protection is triggered. I believe a few weeks ago Chrome was triggering this too, but it looks like a recent update may have resolved it. This is on Tomcat 9.0.45 (though it applies to earlier versions too) using the org.apache.coyote.http11.Http11NioProtocol connector with upgrade to org.apache.coyote.http2.Http2Protocol. With http2 debug logging enabled, there are occasional Payload size [1] interspersed between larger payloads, though it appears to be a couple of back-to-back Payload size [1]'s which may be triggering the org.apache.coyote.http2.ConnectionException: Connection [0], Too much overhead so the connection will be closed. I've found earlier Tomcat conversations which seem to indicate it's a client issue. I've searched through the Firefox bug tracker but have not found anything that looks like this (feels like looking for a needle in a haystack). Is this a known issue in Tomcat or Firefox? It can be resolved by disabling the protection or potentially just tweaking the config values, but if it's a known Firefox issue I'd like to know if something has been raised on their bug tracker I can follow? Nothing in the tracker that I know of off-hand, but there is this informative answer on SO by markt which may help you a little: https://stackoverflow.com/a/61454503/276232 Using the information in that answer may allow you to reconfigure your server to allow Firefox's behavior. It's probably worth us taking some time to adapt markt's SO answer there into a whole section on "Protocol Abuse and Protection Features" in the HTTP/2 configuration guide. There is an open issue for Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=1000809 ACK but there hasn't been much movement. I'll add something to my TODO list to see what can be done to avoid false positives like this as it looks like short sequences of small packets can occur depending on how the client buffers data. Maybe a tweak to the default settings can improve things for "the status quo" for current web browser behavior. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Unable to parse forwarded port issue
Shreya, On 5/6/21 09:45, shreya hegde wrote: Hi Team I have been facing issues lately when a spring boot application acts as a proxy (using embedded tomcat 9.4) passes the request from load balancer to the end web server(jetty) However we see that get requests to through without any error but post request fails with Unable to parse port in forwarded type headers please use forwardheaderfilter with remove only true. Can some one please assist on this. Weird why get is passing and not post This looks like a Spring issue and not a Tomcat one. The message suggests using ForwardHeaderFilter which is Spring-related and not from Tomcat. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat (catalina.jar) Security Question
On 06/05/2021 14:09, Robert Hicks wrote: We are getting evaluated and one of the items that I need to do is change the "ServerInfo.properties" in the catalina.jar to set "server.info" and "server.version" to nonsense (really). I have the following Valve setup as well: At what point would the "ServerInfo.properties" actually show a version and server name to an end user? I am just wondering if mucking with the jar every release is a worthwhile thing and what security implications (if any) are involved. No need to edit the JAR. Extract ServerInfo.properties to $CATALINA_BASE/lib/org/apache/catalina/util and edit the extracted file. It will be used in preference to the one in the JAR. ServerInfo is exposed via ServletContext.getServerInfo() so it is possible that an application will expose it. The DefaultServlet will show it by default if listings are enabled (can be disabled). The ErrorReportValve will show it by default on error pages (can be disabled). The security argument goes something like: "If you expose the software name and version number it makes it easier for an attacker to identify known vulnerabilities for that version and target your server" My personal counter argument goes something like: "Whether you expose the version number or not, if you run a version with a known vulnerability that your are affected by then you are vulnerable. Rather than waste time hiding the version number which is simply security by obscurity - ie no security at all, spend that time doing something useful like upgrading the server so you are no longer exposed to the vulnerability." HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Unable to parse forwarded port issue
Hi Team I have been facing issues lately when a spring boot application acts as a proxy (using embedded tomcat 9.4) passes the request from load balancer to the end web server(jetty) However we see that get requests to through without any error but post request fails with Unable to parse port in forwarded type headers please use forwardheaderfilter with remove only true. Can some one please assist on this. Weird why get is passing and not post Shreya Hegde
Re: Firefox triggers HTTP2 overhead protection - known issue?
On 06/05/2021 13:33, Christopher Schultz wrote: Leon, On 5/6/21 06:25, Leon Atherton wrote: We are seeing that Firefox triggers the HTTP2 overhead protection with multipart file uploads. About 1MB is uploaded before overhead protection is triggered. I believe a few weeks ago Chrome was triggering this too, but it looks like a recent update may have resolved it. This is on Tomcat 9.0.45 (though it applies to earlier versions too) using the org.apache.coyote.http11.Http11NioProtocol connector with upgrade to org.apache.coyote.http2.Http2Protocol. With http2 debug logging enabled, there are occasional Payload size [1] interspersed between larger payloads, though it appears to be a couple of back-to-back Payload size [1]'s which may be triggering the org.apache.coyote.http2.ConnectionException: Connection [0], Too much overhead so the connection will be closed. I've found earlier Tomcat conversations which seem to indicate it's a client issue. I've searched through the Firefox bug tracker but have not found anything that looks like this (feels like looking for a needle in a haystack). Is this a known issue in Tomcat or Firefox? It can be resolved by disabling the protection or potentially just tweaking the config values, but if it's a known Firefox issue I'd like to know if something has been raised on their bug tracker I can follow? Nothing in the tracker that I know of off-hand, but there is this informative answer on SO by markt which may help you a little: https://stackoverflow.com/a/61454503/276232 Using the information in that answer may allow you to reconfigure your server to allow Firefox's behavior. It's probably worth us taking some time to adapt markt's SO answer there into a whole section on "Protocol Abuse and Protection Features" in the HTTP/2 configuration guide. There is an open issue for Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=1000809 but there hasn't been much movement. I'll add something to my TODO list to see what can be done to avoid false positives like this as it looks like short sequences of small packets can occur depending on how the client buffers data. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: temp folder?
What is the setting for unpackWARs for Host? Running directly from a WAR (with unpackWARs="false" file will impact performance. It looks as if something is unpacking the WAR to the temp directory. Tomcat does provide the org.apache.catalina.webresources.ExtractingRoot resources implementation to help alleviate performance issues in this case but that should only extract the JARs in WEB-INF/lib and location they are extracted to should be under the work directory and include "application-jars" in the path. Maybe some custom "unpack to temp" code? Mark On 05/05/2021 20:04, Berneburg, Cris J. - US wrote: Hi Folks Sometimes we get strange errors after deployments to our test server. We just "solved" some weirdness by manually cleaning out the TC temp folder(s) - again. Googling confirms what I thought about the TC work versus temp folder: * "work stores compiled JSPs and other assets". * "temp is used to store files created using the Java File API for creating temporary files". Looking in our TC temp folder, I see subfolders that match all the webapps (some names changed to protect the not-so-innocent): * 0-app1 * 1-app2 * 2-app3 * 3-app4 * 4-docs * 5-manager * 6-trap Looking in a subfolder, like temp/3-app4, it appears to be an exact copy of everything in the webapps/app4 folder, which is just the extracted app4.war file. (The webapps folder has a copy of app4.war.) The temp/app4 folder does not seem to contain temporary files, like output files for Excel reports, etc. Same for the other subfolders. Is that normal? Some technical specifics: * TC 8.5.63 * Java 1.8.0_291 * AWS EC2 instance. * Windows Server 2016. * Instance started as Windows Service. * -Dcatalina.home=D:\Tomcat8_1 * -Dcatalina.base=D:\Tomcat8_1 * -Djava.io.tmpdir=D:\Tomcat8_1\temp * There are other TC instances on the same server. * Each TC instance has multiple apps. I see references to the temp folder in tomcat8-stdout.x.log Below are some excerpts. Why is it trying to access files in the temp subfolder instead of the webapps subfolder? (Looks like I have some app debugging to do?) * 2021-05-05 07:03:38,383 DEBUG [localhost-startStop-1] (?:?) - Attempting to obtain an input stream to file:/D:/Tomcat8_1/temp/0-app1/WEB-INF/classes/action.properties. * 2021-05-05 07:04:52,426 localhost-startStop-1 DEBUG Apache Log4j Core 2.12.1 initializing configuration XmlConfiguration[location=D:\Tomcat8_1\temp\1-app2\WEB-INF\classes\log4j2.xml] * 07:04:53.990 [localhost-startStop-1] DEBUG org.springframework.context.annotation.ClassPathBeanDefinitionScanner - Identified candidate component class: file [D:\Tomcat8_1\temp\1-app2\WEB-INF\classes\app\HelloWorld.class] * 2021-05-05 07:05:10,007 DEBUG [localhost-startStop-1] (?:?) - Attempting to obtain an input stream to file:/D:/Tomcat8_1/temp/2-app3/WEB-INF/classes/action.properties. -- Cris Berneburg CACI Senior Software Engineer This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat (catalina.jar) Security Question
We are getting evaluated and one of the items that I need to do is change the "ServerInfo.properties" in the catalina.jar to set "server.info" and "server.version" to nonsense (really). I have the following Valve setup as well: At what point would the "ServerInfo.properties" actually show a version and server name to an end user? I am just wondering if mucking with the jar every release is a worthwhile thing and what security implications (if any) are involved. Thanks, Bob
Re: Firefox triggers HTTP2 overhead protection - known issue?
Leon, On 5/6/21 06:25, Leon Atherton wrote: We are seeing that Firefox triggers the HTTP2 overhead protection with multipart file uploads. About 1MB is uploaded before overhead protection is triggered. I believe a few weeks ago Chrome was triggering this too, but it looks like a recent update may have resolved it. This is on Tomcat 9.0.45 (though it applies to earlier versions too) using the org.apache.coyote.http11.Http11NioProtocol connector with upgrade to org.apache.coyote.http2.Http2Protocol. With http2 debug logging enabled, there are occasional Payload size [1] interspersed between larger payloads, though it appears to be a couple of back-to-back Payload size [1]'s which may be triggering the org.apache.coyote.http2.ConnectionException: Connection [0], Too much overhead so the connection will be closed. I've found earlier Tomcat conversations which seem to indicate it's a client issue. I've searched through the Firefox bug tracker but have not found anything that looks like this (feels like looking for a needle in a haystack). Is this a known issue in Tomcat or Firefox? It can be resolved by disabling the protection or potentially just tweaking the config values, but if it's a known Firefox issue I'd like to know if something has been raised on their bug tracker I can follow? Nothing in the tracker that I know of off-hand, but there is this informative answer on SO by markt which may help you a little: https://stackoverflow.com/a/61454503/276232 Using the information in that answer may allow you to reconfigure your server to allow Firefox's behavior. It's probably worth us taking some time to adapt markt's SO answer there into a whole section on "Protocol Abuse and Protection Features" in the HTTP/2 configuration guide. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Firefox triggers HTTP2 overhead protection - known issue?
We are seeing that Firefox triggers the HTTP2 overhead protection with multipart file uploads. About 1MB is uploaded before overhead protection is triggered. I believe a few weeks ago Chrome was triggering this too, but it looks like a recent update may have resolved it. This is on Tomcat 9.0.45 (though it applies to earlier versions too) using the org.apache.coyote.http11.Http11NioProtocol connector with upgrade to org.apache.coyote.http2.Http2Protocol. With http2 debug logging enabled, there are occasional Payload size [1] interspersed between larger payloads, though it appears to be a couple of back-to-back Payload size [1]'s which may be triggering the org.apache.coyote.http2.ConnectionException: Connection [0], Too much overhead so the connection will be closed. I've found earlier Tomcat conversations which seem to indicate it's a client issue. I've searched through the Firefox bug tracker but have not found anything that looks like this (feels like looking for a needle in a haystack). Is this a known issue in Tomcat or Firefox? It can be resolved by disabling the protection or potentially just tweaking the config values, but if it's a known Firefox issue I'd like to know if something has been raised on their bug tracker I can follow? Thanks Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
