Re: Tomcat closes connections on unexpected status codes
Chris, On Fri, Apr 19, 2024 at 4:40 AM Christopher Schultz wrote: > > Pawel, > > On 4/18/24 20:21, Pawel Veselov wrote: > >> On 18/04/2024 15:18, Stefan Ansing wrote: > >>> Hi Rémy, Mark, > >>> I just want to make sure that we’re understanding each other. I can see > >>> that the connection needs to be closed in certain conditions to prevent > >>> request smuggling attacks. I certainly don’t want to change that > >>> behaviour. > >>> However, I’m facing a scenario where an application is responding to a > >>> valid request (from HTTP perspective), with a valid response using these > >>> status codes (more specifically status codes 400 and 500). > >> If the request is a valid HTTP request then a 400 status doesn't seem > >> appropriate to me. > > > > It's by now, however, a de-facto standard. Every time I try to > > determine "which HTTP response should I send back in case of issues > > with the data", I find myself scrolling through the list of defined > > codes and not finding anything that would otherwise fit. The HTTP > > spec states what should the server do in case of HTTP protocol errors > > (respond with an appropriate 4xx), but that's all that the spec > > covers for the protocol, and it doesn't prohibit use of 400 for > > application-level errors. Out of the entire 4xx codes, 400 is (maybe > > also 414?) the only one that is used for protocol problems, others > > are for application level errors, but they are very specific and > > limiting (IMHO). > When you say "protocol problems", what protocol are you referring to? In this instance, I meant the actual problems with the HTTP protocol contents. 400 is normally returned by the container because it couldn't understand the HTTP. 414 is probably as well (because container ran out of maximum available space to read in the URI). The rest of the error codes are expected to be produced by the application, but they were devised for that application being a web server. > Because if the request is readable and syntactically correct, there is > no protocol problem. Everything else is ... something else. If you are > establishing a protocol ON TOP OF HTTP then it's a violation of whatever > protocol THAT is, not HTTP. So it's better to return { "error" : "Foo > Protocol violation" } with an appropriate HTTP status code, possibly > even *200*. Yes. But (again, assuming web-service-like implementation), the application can return other HTTP codes for other data (on top of HTTP) issues, i.e., when an endpoint is not found, 404, when data type is wrong - 415; there are also those exuberant codes like 409 and 410. So, on the surface, it looks like the application can use 4xx codes to signal an error, so one can lay down a contract that says "2xx means OK response from the application's perspective, and if there is an error, we'll return some 4xx code", but there *was* (see below) no good 4xx code for saying "Hey, you forgot a JSON property in your input". You're right, it probably should have been 2xx, but it's also cringy to say things like "there are some errors that we will return 4xx for, but for some it will be 2xx, and here is how you differentiate successful 2xx from error response 2xx". I did do that in at least one w/s implementation, and had the customer yell at me for my trouble. This path is also not well supported by standards like OAS/RAML (last time I checked, at least). The description of the responses becomes significantly more complicated in this case. And, again, there is AWS that is a rather large supplier of web services, and they did decide to use 400 for this. The web-service client still needs to distinguish between "400 because the header is mangled", or "400 because a JSON property was missing". The RFC doesn't explicitly limit the use of 400 to HTTP protocol problems, even though all the examples that are listed in the spec *are* such. I did look again at the specs just now, and lo and behold, there is now a code 422. Which, at least on the surface, looks like exactly the code to use for responding with application-related content problems. It wasn't there in RFC#7231 (and neither in #2616 or #2068), but #9110 does have it (yet the language around 400 is still overly broad), which makes it about 2 years old. Now, of course, existing w/s contracts aren't likely to switch to using that, but for any new ones I'll certainly consider it. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
> Assuming it's easy for Tomcat to differentiate between errors generated My PR was based on the assumption that it is easy, since Tomcat always invokes this method[1] if it's a badRequest. [1] https://github.com/apache/tomcat/blob/main/java/org/apache/coyote/http11/Http11Processor.java#L849-L850 On Wed, Apr 24, 2024 at 11:51 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > Stefan, > > On 4/24/24 13:58, Stefan Ansing wrote: > > Op do 18 apr 2024 om 17:42 schreef Mark Thomas : > > > >> On 18/04/2024 15:18, Stefan Ansing wrote: > >>> Hi Rémy, Mark, > >>> > >>> > >>> > >>> I just want to make sure that we’re understanding each other. I can see > >>> that the connection needs to be closed in certain conditions to prevent > >>> request smuggling attacks. I certainly don’t want to change that > >> behaviour. > >>> > >>> However, I’m facing a scenario where an application is responding to a > >>> valid request (from HTTP perspective), with a valid response using > these > >>> status codes (more specifically status codes 400 and 500). > >> > >> If the request is a valid HTTP request then a 400 status doesn't seem > >> appropriate to me. > >> > >> If the server is correctly handling that request to generate the > >> response, a 500 status doesn't seem right either. > >> > >>> > >>> I don’t think that in this scenario a request smuggling attack could be > >>> executed, or am I missing something? > >> > >> The main issue is if the original request is invalid HTTP there is no > >> way to determine where the next HTTP request starts. > >> > >> If there is a proxy in the mix then the risks of something going wrong > >> tend to go up. > >> > >> Mark > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > Hi Mark, > > > > I can see your point of view regarding the use of the status codes for > > application errors. Unfortunately the HTTP spec doesn't clearly > > differentiate the use of status codes for protocol or application errors. > > Which is probably why these status codes are now also commonly used for > > application errors. > > > > Tomcat (and other servlet containers) currently allow applications to set > > any status code, but with the current behaviour of Tomcat this leads to > > unexpected side effects for some status codes. > > > > This behaviour makes it so that Tomcat might not be fit for our purpose > > (Spring Boot services to services communications). I think the way to > > resolve that is to alter the behaviour in Tomcat to differentiate between > > protocol and application errors when using these status codes (and to > make > > this behaviour potentially configurable). I also think that this change > > would benefit most users of Tomcat since the behaviour in this scenario > is > > unnecessary. Would the Tomcat developers be willing to do that? > > Assuming it's easy for Tomcat to differentiate between errors generated > by Tomcat (e.g. "real" 400 responses) and those generated by the > application, I think this is a good idea. HTTP 400 indicates a protocol > error, but if the application is generating it, then Tomcat need not > close the connection. > > Theoretically this could also be true for other status codes as well. I > chose 400 because it means the connection MUST be closed for security if > Tomcat is the one detecting that there is actually an HTTP protocol > violation. > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat closes connections on unexpected status codes
Stefan, On 4/24/24 13:58, Stefan Ansing wrote: Op do 18 apr 2024 om 17:42 schreef Mark Thomas : On 18/04/2024 15:18, Stefan Ansing wrote: Hi Rémy, Mark, I just want to make sure that we’re understanding each other. I can see that the connection needs to be closed in certain conditions to prevent request smuggling attacks. I certainly don’t want to change that behaviour. However, I’m facing a scenario where an application is responding to a valid request (from HTTP perspective), with a valid response using these status codes (more specifically status codes 400 and 500). If the request is a valid HTTP request then a 400 status doesn't seem appropriate to me. If the server is correctly handling that request to generate the response, a 500 status doesn't seem right either. I don’t think that in this scenario a request smuggling attack could be executed, or am I missing something? The main issue is if the original request is invalid HTTP there is no way to determine where the next HTTP request starts. If there is a proxy in the mix then the risks of something going wrong tend to go up. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Hi Mark, I can see your point of view regarding the use of the status codes for application errors. Unfortunately the HTTP spec doesn't clearly differentiate the use of status codes for protocol or application errors. Which is probably why these status codes are now also commonly used for application errors. Tomcat (and other servlet containers) currently allow applications to set any status code, but with the current behaviour of Tomcat this leads to unexpected side effects for some status codes. This behaviour makes it so that Tomcat might not be fit for our purpose (Spring Boot services to services communications). I think the way to resolve that is to alter the behaviour in Tomcat to differentiate between protocol and application errors when using these status codes (and to make this behaviour potentially configurable). I also think that this change would benefit most users of Tomcat since the behaviour in this scenario is unnecessary. Would the Tomcat developers be willing to do that? Assuming it's easy for Tomcat to differentiate between errors generated by Tomcat (e.g. "real" 400 responses) and those generated by the application, I think this is a good idea. HTTP 400 indicates a protocol error, but if the application is generating it, then Tomcat need not close the connection. Theoretically this could also be true for other status codes as well. I chose 400 because it means the connection MUST be closed for security if Tomcat is the one detecting that there is actually an HTTP protocol violation. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
Op do 18 apr 2024 om 17:42 schreef Mark Thomas : > On 18/04/2024 15:18, Stefan Ansing wrote: > > Hi Rémy, Mark, > > > > > > > > I just want to make sure that we’re understanding each other. I can see > > that the connection needs to be closed in certain conditions to prevent > > request smuggling attacks. I certainly don’t want to change that > behaviour. > > > > However, I’m facing a scenario where an application is responding to a > > valid request (from HTTP perspective), with a valid response using these > > status codes (more specifically status codes 400 and 500). > > If the request is a valid HTTP request then a 400 status doesn't seem > appropriate to me. > > If the server is correctly handling that request to generate the > response, a 500 status doesn't seem right either. > > > > > I don’t think that in this scenario a request smuggling attack could be > > executed, or am I missing something? > > The main issue is if the original request is invalid HTTP there is no > way to determine where the next HTTP request starts. > > If there is a proxy in the mix then the risks of something going wrong > tend to go up. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > Hi Mark, I can see your point of view regarding the use of the status codes for application errors. Unfortunately the HTTP spec doesn't clearly differentiate the use of status codes for protocol or application errors. Which is probably why these status codes are now also commonly used for application errors. Tomcat (and other servlet containers) currently allow applications to set any status code, but with the current behaviour of Tomcat this leads to unexpected side effects for some status codes. This behaviour makes it so that Tomcat might not be fit for our purpose (Spring Boot services to services communications). I think the way to resolve that is to alter the behaviour in Tomcat to differentiate between protocol and application errors when using these status codes (and to make this behaviour potentially configurable). I also think that this change would benefit most users of Tomcat since the behaviour in this scenario is unnecessary. Would the Tomcat developers be willing to do that? Stefan
Re: Tomcat closes connections on unexpected status codes
Any chance someone took a look at the PR? Do you guys think this is a viable solution? On Sun, Apr 21, 2024 at 12:54 PM Adwait Kumar Singh wrote: > https://github.com/apache/tomcat/pull/723 is a draft PR of the idea I was > talking about earlier, i.e close the connection on a bad request but > otherwise allow it to be configurable by the user. > > Currently you need to subclass Http11Processor and override > statusDropsConnections, but this is just to get early feedback. If people > are aligned with this approach, I can polish the PR (add tests, make the > status code list actually configurable etc etc) > > On Fri, Apr 19, 2024 at 5:51 AM Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> Harri, >> >> On 4/19/24 08:10, Harri Pesonen wrote: >> > I have developed a restful web service, which uses HTTP response codes >> 200 OK, 201 Created, 204 No Content and 404 Not Found. >> > It does not use 400 Bad Request or 500 Internal Server Error normally. >> > 400 Bad Request is more common than 500 Internal Server Error, which >> should basically never happen. >> >> I disagree. There aren't many good 5xx errors, so sometimes "500 Oops" >> is the best you can do. HTTP 5xx doesn't even really allow you to >> differentiate between "server error" and "application error". >> >> > 400 Bad Request is the best response in many cases, if client gives >> some query parameter which is not supported by the application logic. >> >> -1 >> >> 400 means that the HTTP request is bad. In your situation, the HTTP >> request is 100% valid. It's your application that is saying "sorry, >> client, you did something we don't allow". This is an application-level >> error, not a protocol-level error. To me, 400 means "HTTP protocol >> error" and should never be substituted for "application protocol error". >> >> > I think that it would be better not to close connection in this case, >> if the error comes from application. >> > I wonder if there is option for application to define that connection >> should be closed or not after the response has been sent? >> > Or is the option only from the client. >> > >> > For me this 404 Not Found is also a small problem, as it is error, but >> it can happen quite often. >> > HTTP errors are not nice in logs. >> > Normally if you try to fetch some restful resource, which does not >> exist, then it returns 404 Not Found. >> > GET /service/resource/id => 404 Not Found >> > If I now had an option to rewrite the service, I would probably use 204 >> No Content in this case as well, to avoid errors. >> >> 404 and 204 are fundamentally different responses. 404 means "this >> resource does not exist" and 204 means "this resource DOES EXIST but it >> doesn't contain anything". Your application may not differentiate >> between those two cases, but as a client I would be confused if "Not >> Found" was replaced by "Found to be empty" in all cases. >> >> > 204 No Content is normally used with PUT and DELETE requests. >> >> Yes, you can use those. 200 would also make sense and, of course 201 for >> new resources. >> >> -chris >> >> > -Original Message- >> > From: Christopher Schultz >> > Sent: perjantai 19. huhtikuuta 2024 14.27 >> > To: users@tomcat.apache.org >> > Subject: Re: Tomcat closes connections on unexpected status codes >> > >> > Mark, >> > >> > On 4/18/24 11:38, Mark Thomas wrote: >> >> On 18/04/2024 15:16, Adwait Kumar Singh wrote: >> >>> I think we should *always* close connections in cases where it can >> >>> lead to request smuggling vulnerabilities like when there is an error >> >>> during header or request line parsing, but allowing the user to >> >>> control connection close when the status is being set by the user, >> >>> should be safe? >> >> >> >> I'm not (yet) convinced distinguishing between those scenarios is >> >> always going to be possible. >> >> >> >>> It allows users to send back responses like InvalidInputException >> >>> with a >> >>> 400 status without being forced to close the connection. >> >> >> >> I appreciate why a 400 is used here but 400 has always struck me as >> >> more for protocol level issues rather than application level issues. >> > >> > Didn't s
Re: Tomcat closes connections on unexpected status codes
https://github.com/apache/tomcat/pull/723 is a draft PR of the idea I was talking about earlier, i.e close the connection on a bad request but otherwise allow it to be configurable by the user. Currently you need to subclass Http11Processor and override statusDropsConnections, but this is just to get early feedback. If people are aligned with this approach, I can polish the PR (add tests, make the status code list actually configurable etc etc) On Fri, Apr 19, 2024 at 5:51 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > Harri, > > On 4/19/24 08:10, Harri Pesonen wrote: > > I have developed a restful web service, which uses HTTP response codes > 200 OK, 201 Created, 204 No Content and 404 Not Found. > > It does not use 400 Bad Request or 500 Internal Server Error normally. > > 400 Bad Request is more common than 500 Internal Server Error, which > should basically never happen. > > I disagree. There aren't many good 5xx errors, so sometimes "500 Oops" > is the best you can do. HTTP 5xx doesn't even really allow you to > differentiate between "server error" and "application error". > > > 400 Bad Request is the best response in many cases, if client gives some > query parameter which is not supported by the application logic. > > -1 > > 400 means that the HTTP request is bad. In your situation, the HTTP > request is 100% valid. It's your application that is saying "sorry, > client, you did something we don't allow". This is an application-level > error, not a protocol-level error. To me, 400 means "HTTP protocol > error" and should never be substituted for "application protocol error". > > > I think that it would be better not to close connection in this case, if > the error comes from application. > > I wonder if there is option for application to define that connection > should be closed or not after the response has been sent? > > Or is the option only from the client. > > > > For me this 404 Not Found is also a small problem, as it is error, but > it can happen quite often. > > HTTP errors are not nice in logs. > > Normally if you try to fetch some restful resource, which does not > exist, then it returns 404 Not Found. > > GET /service/resource/id => 404 Not Found > > If I now had an option to rewrite the service, I would probably use 204 > No Content in this case as well, to avoid errors. > > 404 and 204 are fundamentally different responses. 404 means "this > resource does not exist" and 204 means "this resource DOES EXIST but it > doesn't contain anything". Your application may not differentiate > between those two cases, but as a client I would be confused if "Not > Found" was replaced by "Found to be empty" in all cases. > > > 204 No Content is normally used with PUT and DELETE requests. > > Yes, you can use those. 200 would also make sense and, of course 201 for > new resources. > > -chris > > > -Original Message- > > From: Christopher Schultz > > Sent: perjantai 19. huhtikuuta 2024 14.27 > > To: users@tomcat.apache.org > > Subject: Re: Tomcat closes connections on unexpected status codes > > > > Mark, > > > > On 4/18/24 11:38, Mark Thomas wrote: > >> On 18/04/2024 15:16, Adwait Kumar Singh wrote: > >>> I think we should *always* close connections in cases where it can > >>> lead to request smuggling vulnerabilities like when there is an error > >>> during header or request line parsing, but allowing the user to > >>> control connection close when the status is being set by the user, > >>> should be safe? > >> > >> I'm not (yet) convinced distinguishing between those scenarios is > >> always going to be possible. > >> > >>> It allows users to send back responses like InvalidInputException > >>> with a > >>> 400 status without being forced to close the connection. > >> > >> I appreciate why a 400 is used here but 400 has always struck me as > >> more for protocol level issues rather than application level issues. > > > > Didn't someone tell me recently that, technically, ANY client-error is > allowed to trigger a 400 response without being more specific? > > > >> That is the fundamental problem here. The status codes are being used > >> for two completely different purposes. > > > > +1 > > > > I've always found it distasteful when REST services do this. To me, 400 > means "the request was actually malformed". If you need authentication, > that's a 401. If you aren't allowed,
Re: Tomcat closes connections on unexpected status codes
Harri, On 4/19/24 08:10, Harri Pesonen wrote: I have developed a restful web service, which uses HTTP response codes 200 OK, 201 Created, 204 No Content and 404 Not Found. It does not use 400 Bad Request or 500 Internal Server Error normally. 400 Bad Request is more common than 500 Internal Server Error, which should basically never happen. I disagree. There aren't many good 5xx errors, so sometimes "500 Oops" is the best you can do. HTTP 5xx doesn't even really allow you to differentiate between "server error" and "application error". 400 Bad Request is the best response in many cases, if client gives some query parameter which is not supported by the application logic. -1 400 means that the HTTP request is bad. In your situation, the HTTP request is 100% valid. It's your application that is saying "sorry, client, you did something we don't allow". This is an application-level error, not a protocol-level error. To me, 400 means "HTTP protocol error" and should never be substituted for "application protocol error". I think that it would be better not to close connection in this case, if the error comes from application. I wonder if there is option for application to define that connection should be closed or not after the response has been sent? Or is the option only from the client. For me this 404 Not Found is also a small problem, as it is error, but it can happen quite often. HTTP errors are not nice in logs. Normally if you try to fetch some restful resource, which does not exist, then it returns 404 Not Found. GET /service/resource/id => 404 Not Found If I now had an option to rewrite the service, I would probably use 204 No Content in this case as well, to avoid errors. 404 and 204 are fundamentally different responses. 404 means "this resource does not exist" and 204 means "this resource DOES EXIST but it doesn't contain anything". Your application may not differentiate between those two cases, but as a client I would be confused if "Not Found" was replaced by "Found to be empty" in all cases. 204 No Content is normally used with PUT and DELETE requests. Yes, you can use those. 200 would also make sense and, of course 201 for new resources. -chris -Original Message- From: Christopher Schultz Sent: perjantai 19. huhtikuuta 2024 14.27 To: users@tomcat.apache.org Subject: Re: Tomcat closes connections on unexpected status codes Mark, On 4/18/24 11:38, Mark Thomas wrote: On 18/04/2024 15:16, Adwait Kumar Singh wrote: I think we should *always* close connections in cases where it can lead to request smuggling vulnerabilities like when there is an error during header or request line parsing, but allowing the user to control connection close when the status is being set by the user, should be safe? I'm not (yet) convinced distinguishing between those scenarios is always going to be possible. It allows users to send back responses like InvalidInputException with a 400 status without being forced to close the connection. I appreciate why a 400 is used here but 400 has always struck me as more for protocol level issues rather than application level issues. Didn't someone tell me recently that, technically, ANY client-error is allowed to trigger a 400 response without being more specific? That is the fundamental problem here. The status codes are being used for two completely different purposes. +1 I've always found it distasteful when REST services do this. To me, 400 means "the request was actually malformed". If you need authentication, that's a 401. If you aren't allowed, that's 403. If you didn't provide a required header, that's a 412, etc. I've usually found the "correct" response code to use for every situation and I've never written an application that returns a 400 response directly. -chris On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat wrote: On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: On 18/04/2024 09:07, Stefan Ansing wrote: Hi, We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19) where we see that HTTP/1.1 connections are closed whenever a servlet application returns the following status codes: 400, 408, 411, 414, 500, 503, 501. This causes client applications to rapidly reconnect and induce high server-side CPU load due to doing TLS handshakes. The 400 and 500 status codes are commonly used in RESTful microservices to communicate errors. Reviewing RFC 9112 I couldn't find any requirement for closing connections on these status codes. After testing with Undertow (version 2.3.12), where we didn't see the same behaviour, we believe that these status codes do not necessitate a new connection. The Tomcat developers disagree. Connections are closed after these status codes to avoid various forms of request smuggling attacks. Checking the Tomcat
RE: Tomcat closes connections on unexpected status codes
I have developed a restful web service, which uses HTTP response codes 200 OK, 201 Created, 204 No Content and 404 Not Found. It does not use 400 Bad Request or 500 Internal Server Error normally. 400 Bad Request is more common than 500 Internal Server Error, which should basically never happen. 400 Bad Request is the best response in many cases, if client gives some query parameter which is not supported by the application logic. I think that it would be better not to close connection in this case, if the error comes from application. I wonder if there is option for application to define that connection should be closed or not after the response has been sent? Or is the option only from the client. For me this 404 Not Found is also a small problem, as it is error, but it can happen quite often. HTTP errors are not nice in logs. Normally if you try to fetch some restful resource, which does not exist, then it returns 404 Not Found. GET /service/resource/id => 404 Not Found If I now had an option to rewrite the service, I would probably use 204 No Content in this case as well, to avoid errors. 204 No Content is normally used with PUT and DELETE requests. -Harri -Original Message- From: Christopher Schultz Sent: perjantai 19. huhtikuuta 2024 14.27 To: users@tomcat.apache.org Subject: Re: Tomcat closes connections on unexpected status codes Mark, On 4/18/24 11:38, Mark Thomas wrote: > On 18/04/2024 15:16, Adwait Kumar Singh wrote: >> I think we should *always* close connections in cases where it can >> lead to request smuggling vulnerabilities like when there is an error >> during header or request line parsing, but allowing the user to >> control connection close when the status is being set by the user, >> should be safe? > > I'm not (yet) convinced distinguishing between those scenarios is > always going to be possible. > >> It allows users to send back responses like InvalidInputException >> with a >> 400 status without being forced to close the connection. > > I appreciate why a 400 is used here but 400 has always struck me as > more for protocol level issues rather than application level issues. Didn't someone tell me recently that, technically, ANY client-error is allowed to trigger a 400 response without being more specific? > That is the fundamental problem here. The status codes are being used > for two completely different purposes. +1 I've always found it distasteful when REST services do this. To me, 400 means "the request was actually malformed". If you need authentication, that's a 401. If you aren't allowed, that's 403. If you didn't provide a required header, that's a 412, etc. I've usually found the "correct" response code to use for every situation and I've never written an application that returns a 400 response directly. -chris >> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat wrote: >> >>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: >>>> >>>> On 18/04/2024 09:07, Stefan Ansing wrote: >>>>> Hi, >>>>> >>>>> We've observed some unexpected behaviour in Apache Tomcat (version >>> 10.1.19) >>>>> where we see that HTTP/1.1 connections are closed whenever a >>>>> servlet application returns the following status codes: 400, 408, >>>>> 411, 414, >>> 500, >>>>> 503, 501. This causes client applications to rapidly reconnect and >>> induce >>>>> high server-side CPU load due to doing TLS handshakes. >>>>> >>>>> The 400 and 500 status codes are commonly used in RESTful >>> microservices to >>>>> communicate errors. Reviewing RFC 9112 I couldn't find any >>>>> requirement >>> for >>>>> closing connections on these status codes. >>>>> >>>>> After testing with Undertow (version 2.3.12), where we didn't see >>>>> the >>> same >>>>> behaviour, we believe that these status codes do not necessitate a >>>>> new connection. >>>> >>>> The Tomcat developers disagree. Connections are closed after these >>>> status codes to avoid various forms of request smuggling attacks. >>>> >>>>> Checking the Tomcat sources makes me believe that the behaviour is >>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of >>>>> status codes and to discuss the possibilities of making the >>>>> behaviour >>> configurable. >>>> >>>> Making this list of status codes configurable seems reasonable. The >>>> default can stay as current an
Re: Tomcat closes connections on unexpected status codes
Pawel, On 4/18/24 20:21, Pawel Veselov wrote: On 18/04/2024 15:18, Stefan Ansing wrote: Hi Rémy, Mark, I just want to make sure that we’re understanding each other. I can see that the connection needs to be closed in certain conditions to prevent request smuggling attacks. I certainly don’t want to change that behaviour. However, I’m facing a scenario where an application is responding to a valid request (from HTTP perspective), with a valid response using these status codes (more specifically status codes 400 and 500). If the request is a valid HTTP request then a 400 status doesn't seem appropriate to me. It's by now, however, a de-facto standard. Every time I try to determine "which HTTP response should I send back in case of issues with the data", I find myself scrolling through the list of defined codes and not finding anything that would otherwise fit. The HTTP spec states what should the server do in case of HTTP protocol errors (respond with an appropriate 4xx), but that's all that the spec covers for the protocol, and it doesn't prohibit use of 400 for application-level errors. Out of the entire 4xx codes, 400 is (maybe also 414?) the only one that is used for protocol problems, others are for application level errors, but they are very specific and limiting (IMHO). When you say "protocol problems", what protocol are you referring to? Because if the request is readable and syntactically correct, there is no protocol problem. Everything else is ... something else. If you are establishing a protocol ON TOP OF HTTP then it's a violation of whatever protocol THAT is, not HTTP. So it's better to return { "error" : "Foo Protocol violation" } with an appropriate HTTP status code, possibly even *200*. Then, how do I return error paths, when implementing things like a web service? Either returning it with 200 and custom headers that indicate it's an error response, or a 400 with the same. In which case, the difference on the client side is - does it have to logically differentiate valid response from an error based just on these custom headers/mime type, or differentiate protocol-related 400 from application-related 400, and can always treat 200 as proper. Then I look at "what do others do", and I see AWS that uses 400 for all client errors, and the way a client knows it's an application error is because there is content that wouldn't be typically returned if there is a protocol problem. So I follow suit. If the server is correctly handling that request to generate the response, a 500 status doesn't seem right either. Right. 5xx - my fault, 4xx - your fault. I don’t think that in this scenario a request smuggling attack could be executed, or am I missing something? The main issue is if the original request is invalid HTTP there is no way to determine where the next HTTP request starts. I'd imagine it reasonable that as long as Tomcat doesn't find any issues with the HTTP protocol itself - closing connections wouldn't serve any helpful purpose in that case, except for not having to read the remainder of the message. I'd like to know how the connection can be mucked up otherwise, whether a proxy is involved or not. If there is a proxy in the mix then the risks of something going wrong tend to go up. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
Pawel, On 4/18/24 20:32, Pawel Veselov wrote: On Thu, Apr 18, 2024 at 9:40 AM Adwait Kumar Singh wrote: I'm not (yet) convinced distinguishing between those scenarios is always going to be possible. I have a Tomcat patch which we use at work to do this, i.e always close the connection if HTTP parsing fails but not if it's a user set status. I can create a PR for feedback. It can't be that straightforward though. The HTTP parsing can very well fail well past the point the application was invoked, and returned a status, during chunked request parsing. The server can't respond with another error though, as the headers are/can be committed, it has no recourse but to try to send them if possible, and then shutdown the connection. Assuming that the container is still in charge of the HTTP part of things, then the container definitely knows if the request has experienced some problem with e.g. chunked content. Tomcat has private information about requests that it keeps separate from what applications can (usually) get access to, and it could simply mark a request as "requiring connection termination" and doing that at the appropriate time when execution finishes with the application and returns to the container. And handling incomplete body reads from the application is also going to be a nuisance that has to be dealt with. Tomcat will have to consume the remainder of the body without application doing so (Tomcat already does this.) , which may not be desirable in all cases, and will require either an API change to be supported as being signalled by the application (here is my status, but PLEASE don't break the HTTP connection), or requiring the application to diligently consume the body till the end before returning service back to Tomcat, or having that as a configuration parameter. I'm not sure this is different from today. A new API isn't strictly necessary, here. The application can communicate with the container using request attributes. But I'm not sure anything like that is needed. I actually don't know how Tomcat deals with bodies that aren't fully read by the app right now. I would very much like to know for sure. Tomcat does one of two things: 1. Reads the remainder of the request entity and discards it 2. Closes the connection without finishing the read I believe it tries (1) for a while and, after some configurable number of bytes ( maxSwallowSize), it switches to (2) and gives up. -chris On Thu, Apr 18, 2024 at 8:42 AM Mark Thomas wrote: On 18/04/2024 15:18, Stefan Ansing wrote: Hi Rémy, Mark, I just want to make sure that we’re understanding each other. I can see that the connection needs to be closed in certain conditions to prevent request smuggling attacks. I certainly don’t want to change that behaviour. However, I’m facing a scenario where an application is responding to a valid request (from HTTP perspective), with a valid response using these status codes (more specifically status codes 400 and 500). If the request is a valid HTTP request then a 400 status doesn't seem appropriate to me. If the server is correctly handling that request to generate the response, a 500 status doesn't seem right either. I don’t think that in this scenario a request smuggling attack could be executed, or am I missing something? The main issue is if the original request is invalid HTTP there is no way to determine where the next HTTP request starts. If there is a proxy in the mix then the risks of something going wrong tend to go up. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
Mark, On 4/18/24 11:38, Mark Thomas wrote: On 18/04/2024 15:16, Adwait Kumar Singh wrote: I think we should *always* close connections in cases where it can lead to request smuggling vulnerabilities like when there is an error during header or request line parsing, but allowing the user to control connection close when the status is being set by the user, should be safe? I'm not (yet) convinced distinguishing between those scenarios is always going to be possible. It allows users to send back responses like InvalidInputException with a 400 status without being forced to close the connection. I appreciate why a 400 is used here but 400 has always struck me as more for protocol level issues rather than application level issues. Didn't someone tell me recently that, technically, ANY client-error is allowed to trigger a 400 response without being more specific? That is the fundamental problem here. The status codes are being used for two completely different purposes. +1 I've always found it distasteful when REST services do this. To me, 400 means "the request was actually malformed". If you need authentication, that's a 401. If you aren't allowed, that's 403. If you didn't provide a required header, that's a 412, etc. I've usually found the "correct" response code to use for every situation and I've never written an application that returns a 400 response directly. -chris On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat wrote: On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: On 18/04/2024 09:07, Stefan Ansing wrote: Hi, We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19) where we see that HTTP/1.1 connections are closed whenever a servlet application returns the following status codes: 400, 408, 411, 414, 500, 503, 501. This causes client applications to rapidly reconnect and induce high server-side CPU load due to doing TLS handshakes. The 400 and 500 status codes are commonly used in RESTful microservices to communicate errors. Reviewing RFC 9112 I couldn't find any requirement for closing connections on these status codes. After testing with Undertow (version 2.3.12), where we didn't see the same behaviour, we believe that these status codes do not necessitate a new connection. The Tomcat developers disagree. Connections are closed after these status codes to avoid various forms of request smuggling attacks. Checking the Tomcat sources makes me believe that the behaviour is hard-coded[1]. I'm reaching out here to re-evaluate the list of status codes and to discuss the possibilities of making the behaviour configurable. Making this list of status codes configurable seems reasonable. The default can stay as current and if users want to change it then they have to accept the associated security risks. If it's insecure, then it would still be a valid CVE even if the configuration is optional. We don't have an "allowSmuggling" attribute on the connector to relax header or status line parsing, even though many would have wanted it in the past ... Rémy Mark A colleague of mine reported a bug for this issue: https://bz.apache.org/bugzilla/show_bug.cgi?id=68901 Kind regards, Stefan Ansing [1]: https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
All, On 4/18/24 10:16, Adwait Kumar Singh wrote: I think we should *always* close connections in cases where it can lead to request smuggling vulnerabilities like when there is an error during header or request line parsing, but allowing the user to control connection close when the status is being set by the user, should be safe? It allows users to send back responses like InvalidInputException with a 400 status without being forced to close the connection. Maybe the behavior depends upon the source of the response code. If Tomcat itself determines that the connection should be terminated, it should. But if the application is the one returning the 4xx error, Tomcat should not terminate the connection. Does that sound like a decent compromise and is it even possible to differentiate? -chris On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat wrote: On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: On 18/04/2024 09:07, Stefan Ansing wrote: Hi, We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19) where we see that HTTP/1.1 connections are closed whenever a servlet application returns the following status codes: 400, 408, 411, 414, 500, 503, 501. This causes client applications to rapidly reconnect and induce high server-side CPU load due to doing TLS handshakes. The 400 and 500 status codes are commonly used in RESTful microservices to communicate errors. Reviewing RFC 9112 I couldn't find any requirement for closing connections on these status codes. After testing with Undertow (version 2.3.12), where we didn't see the same behaviour, we believe that these status codes do not necessitate a new connection. The Tomcat developers disagree. Connections are closed after these status codes to avoid various forms of request smuggling attacks. Checking the Tomcat sources makes me believe that the behaviour is hard-coded[1]. I'm reaching out here to re-evaluate the list of status codes and to discuss the possibilities of making the behaviour configurable. Making this list of status codes configurable seems reasonable. The default can stay as current and if users want to change it then they have to accept the associated security risks. If it's insecure, then it would still be a valid CVE even if the configuration is optional. We don't have an "allowSmuggling" attribute on the connector to relax header or status line parsing, even though many would have wanted it in the past ... Rémy Mark A colleague of mine reported a bug for this issue: https://bz.apache.org/bugzilla/show_bug.cgi?id=68901 Kind regards, Stefan Ansing [1]: https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
On Thu, Apr 18, 2024 at 9:40 AM Adwait Kumar Singh wrote: > > I'm not (yet) convinced distinguishing between those scenarios is always > > going to be possible. > I have a Tomcat patch which we use at work to do this, i.e always close the > connection if HTTP parsing fails but not if it's a user set status. I can > create a PR for feedback. It can't be that straightforward though. The HTTP parsing can very well fail well past the point the application was invoked, and returned a status, during chunked request parsing. The server can't respond with another error though, as the headers are/can be committed, it has no recourse but to try to send them if possible, and then shutdown the connection. And handling incomplete body reads from the application is also going to be a nuisance that has to be dealt with. Tomcat will have to consume the remainder of the body without application doing so, which may not be desirable in all cases, and will require either an API change to be supported as being signalled by the application (here is my status, but PLEASE don't break the HTTP connection), or requiring the application to diligently consume the body till the end before returning service back to Tomcat, or having that as a configuration parameter. I actually don't know how Tomcat deals with bodies that aren't fully read by the app right now. I would very much like to know for sure. > > On Thu, Apr 18, 2024 at 8:42 AM Mark Thomas wrote: > > > On 18/04/2024 15:18, Stefan Ansing wrote: > > > Hi Rémy, Mark, > > > > > > > > > > > > I just want to make sure that we’re understanding each other. I can see > > > that the connection needs to be closed in certain conditions to prevent > > > request smuggling attacks. I certainly don’t want to change that > > behaviour. > > > > > > However, I’m facing a scenario where an application is responding to a > > > valid request (from HTTP perspective), with a valid response using these > > > status codes (more specifically status codes 400 and 500). > > > > If the request is a valid HTTP request then a 400 status doesn't seem > > appropriate to me. > > > > If the server is correctly handling that request to generate the > > response, a 500 status doesn't seem right either. > > > > > > > > I don’t think that in this scenario a request smuggling attack could be > > > executed, or am I missing something? > > > > The main issue is if the original request is invalid HTTP there is no > > way to determine where the next HTTP request starts. > > > > If there is a proxy in the mix then the risks of something going wrong > > tend to go up. > > > > Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
> On 18/04/2024 15:18, Stefan Ansing wrote: > > Hi Rémy, Mark, > > I just want to make sure that we’re understanding each other. I can see > > that the connection needs to be closed in certain conditions to prevent > > request smuggling attacks. I certainly don’t want to change that behaviour. > > However, I’m facing a scenario where an application is responding to a > > valid request (from HTTP perspective), with a valid response using these > > status codes (more specifically status codes 400 and 500). > If the request is a valid HTTP request then a 400 status doesn't seem > appropriate to me. It's by now, however, a de-facto standard. Every time I try to determine "which HTTP response should I send back in case of issues with the data", I find myself scrolling through the list of defined codes and not finding anything that would otherwise fit. The HTTP spec states what should the server do in case of HTTP protocol errors (respond with an appropriate 4xx), but that's all that the spec covers for the protocol, and it doesn't prohibit use of 400 for application-level errors. Out of the entire 4xx codes, 400 is (maybe also 414?) the only one that is used for protocol problems, others are for application level errors, but they are very specific and limiting (IMHO). Then, how do I return error paths, when implementing things like a web service? Either returning it with 200 and custom headers that indicate it's an error response, or a 400 with the same. In which case, the difference on the client side is - does it have to logically differentiate valid response from an error based just on these custom headers/mime type, or differentiate protocol-related 400 from application-related 400, and can always treat 200 as proper. Then I look at "what do others do", and I see AWS that uses 400 for all client errors, and the way a client knows it's an application error is because there is content that wouldn't be typically returned if there is a protocol problem. So I follow suit. > If the server is correctly handling that request to generate the > response, a 500 status doesn't seem right either. Right. 5xx - my fault, 4xx - your fault. > > I don’t think that in this scenario a request smuggling attack could be > > executed, or am I missing something? > The main issue is if the original request is invalid HTTP there is no > way to determine where the next HTTP request starts. I'd imagine it reasonable that as long as Tomcat doesn't find any issues with the HTTP protocol itself - closing connections wouldn't serve any helpful purpose in that case, except for not having to read the remainder of the message. I'd like to know how the connection can be mucked up otherwise, whether a proxy is involved or not. > > If there is a proxy in the mix then the risks of something going wrong > tend to go up. > > Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
> > I'm not (yet) convinced distinguishing between those scenarios is always > going to be possible. I have a Tomcat patch which we use at work to do this, i.e always close the connection if HTTP parsing fails but not if it's a user set status. I can create a PR for feedback. On Thu, Apr 18, 2024 at 8:42 AM Mark Thomas wrote: > On 18/04/2024 15:18, Stefan Ansing wrote: > > Hi Rémy, Mark, > > > > > > > > I just want to make sure that we’re understanding each other. I can see > > that the connection needs to be closed in certain conditions to prevent > > request smuggling attacks. I certainly don’t want to change that > behaviour. > > > > However, I’m facing a scenario where an application is responding to a > > valid request (from HTTP perspective), with a valid response using these > > status codes (more specifically status codes 400 and 500). > > If the request is a valid HTTP request then a 400 status doesn't seem > appropriate to me. > > If the server is correctly handling that request to generate the > response, a 500 status doesn't seem right either. > > > > > I don’t think that in this scenario a request smuggling attack could be > > executed, or am I missing something? > > The main issue is if the original request is invalid HTTP there is no > way to determine where the next HTTP request starts. > > If there is a proxy in the mix then the risks of something going wrong > tend to go up. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat closes connections on unexpected status codes
On 18/04/2024 15:18, Stefan Ansing wrote: Hi Rémy, Mark, I just want to make sure that we’re understanding each other. I can see that the connection needs to be closed in certain conditions to prevent request smuggling attacks. I certainly don’t want to change that behaviour. However, I’m facing a scenario where an application is responding to a valid request (from HTTP perspective), with a valid response using these status codes (more specifically status codes 400 and 500). If the request is a valid HTTP request then a 400 status doesn't seem appropriate to me. If the server is correctly handling that request to generate the response, a 500 status doesn't seem right either. I don’t think that in this scenario a request smuggling attack could be executed, or am I missing something? The main issue is if the original request is invalid HTTP there is no way to determine where the next HTTP request starts. If there is a proxy in the mix then the risks of something going wrong tend to go up. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
On 18/04/2024 15:16, Adwait Kumar Singh wrote: I think we should *always* close connections in cases where it can lead to request smuggling vulnerabilities like when there is an error during header or request line parsing, but allowing the user to control connection close when the status is being set by the user, should be safe? I'm not (yet) convinced distinguishing between those scenarios is always going to be possible. It allows users to send back responses like InvalidInputException with a 400 status without being forced to close the connection. I appreciate why a 400 is used here but 400 has always struck me as more for protocol level issues rather than application level issues. That is the fundamental problem here. The status codes are being used for two completely different purposes. Mark On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat wrote: On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: On 18/04/2024 09:07, Stefan Ansing wrote: Hi, We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19) where we see that HTTP/1.1 connections are closed whenever a servlet application returns the following status codes: 400, 408, 411, 414, 500, 503, 501. This causes client applications to rapidly reconnect and induce high server-side CPU load due to doing TLS handshakes. The 400 and 500 status codes are commonly used in RESTful microservices to communicate errors. Reviewing RFC 9112 I couldn't find any requirement for closing connections on these status codes. After testing with Undertow (version 2.3.12), where we didn't see the same behaviour, we believe that these status codes do not necessitate a new connection. The Tomcat developers disagree. Connections are closed after these status codes to avoid various forms of request smuggling attacks. Checking the Tomcat sources makes me believe that the behaviour is hard-coded[1]. I'm reaching out here to re-evaluate the list of status codes and to discuss the possibilities of making the behaviour configurable. Making this list of status codes configurable seems reasonable. The default can stay as current and if users want to change it then they have to accept the associated security risks. If it's insecure, then it would still be a valid CVE even if the configuration is optional. We don't have an "allowSmuggling" attribute on the connector to relax header or status line parsing, even though many would have wanted it in the past ... Rémy Mark A colleague of mine reported a bug for this issue: https://bz.apache.org/bugzilla/show_bug.cgi?id=68901 Kind regards, Stefan Ansing [1]: https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
On 18/04/2024 14:41, Rémy Maucherat wrote: On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: On 18/04/2024 09:07, Stefan Ansing wrote: Hi, We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19) where we see that HTTP/1.1 connections are closed whenever a servlet application returns the following status codes: 400, 408, 411, 414, 500, 503, 501. This causes client applications to rapidly reconnect and induce high server-side CPU load due to doing TLS handshakes. The 400 and 500 status codes are commonly used in RESTful microservices to communicate errors. Reviewing RFC 9112 I couldn't find any requirement for closing connections on these status codes. After testing with Undertow (version 2.3.12), where we didn't see the same behaviour, we believe that these status codes do not necessitate a new connection. The Tomcat developers disagree. Connections are closed after these status codes to avoid various forms of request smuggling attacks. Checking the Tomcat sources makes me believe that the behaviour is hard-coded[1]. I'm reaching out here to re-evaluate the list of status codes and to discuss the possibilities of making the behaviour configurable. Making this list of status codes configurable seems reasonable. The default can stay as current and if users want to change it then they have to accept the associated security risks. If it's insecure, then it would still be a valid CVE even if the configuration is optional. We don't have an "allowSmuggling" attribute on the connector to relax header or status line parsing, even though many would have wanted it in the past ... I don't think it is quite that black and white. If the client connects directly to the server (no proxy) then there request smuggling is a lot less likely and arguably impossible (assuming no Tomcat bugs). If there is a proxy that is when there is a risk of request smuggling. Given that there are circumstances where the risk of request smuggling is low (possibly none) it seems reasonable to allow users to configure the status codes that trigger connection close. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
Op do 18 apr 2024 om 15:41 schreef Rémy Maucherat : > On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: > > > > On 18/04/2024 09:07, Stefan Ansing wrote: > > > Hi, > > > > > > We've observed some unexpected behaviour in Apache Tomcat (version > 10.1.19) > > > where we see that HTTP/1.1 connections are closed whenever a servlet > > > application returns the following status codes: 400, 408, 411, 414, > 500, > > > 503, 501. This causes client applications to rapidly reconnect and > induce > > > high server-side CPU load due to doing TLS handshakes. > > > > > > The 400 and 500 status codes are commonly used in RESTful > microservices to > > > communicate errors. Reviewing RFC 9112 I couldn't find any requirement > for > > > closing connections on these status codes. > > > > > > After testing with Undertow (version 2.3.12), where we didn't see the > same > > > behaviour, we believe that these status codes do not necessitate a new > > > connection. > > > > The Tomcat developers disagree. Connections are closed after these > > status codes to avoid various forms of request smuggling attacks. > > > > > Checking the Tomcat sources makes me believe that the behaviour is > > > hard-coded[1]. I'm reaching out here to re-evaluate the list of status > > > codes and to discuss the possibilities of making the behaviour > configurable. > > > > Making this list of status codes configurable seems reasonable. The > > default can stay as current and if users want to change it then they > > have to accept the associated security risks. > > If it's insecure, then it would still be a valid CVE even if the > configuration is optional. We don't have an "allowSmuggling" attribute > on the connector to relax header or status line parsing, even though > many would have wanted it in the past ... > > Rémy > > > Mark > > > > > > > > > > A colleague of mine reported a bug for this issue: > > > https://bz.apache.org/bugzilla/show_bug.cgi?id=68901 > > > > > > Kind regards, > > > Stefan Ansing > > > > > > [1]: > > > > https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617 > > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > Hi Rémy, Mark, I just want to make sure that we’re understanding each other. I can see that the connection needs to be closed in certain conditions to prevent request smuggling attacks. I certainly don’t want to change that behaviour. However, I’m facing a scenario where an application is responding to a valid request (from HTTP perspective), with a valid response using these status codes (more specifically status codes 400 and 500). I don’t think that in this scenario a request smuggling attack could be executed, or am I missing something? Stefan
Re: Tomcat closes connections on unexpected status codes
I think we should *always* close connections in cases where it can lead to request smuggling vulnerabilities like when there is an error during header or request line parsing, but allowing the user to control connection close when the status is being set by the user, should be safe? It allows users to send back responses like InvalidInputException with a 400 status without being forced to close the connection. On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat wrote: > On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: > > > > On 18/04/2024 09:07, Stefan Ansing wrote: > > > Hi, > > > > > > We've observed some unexpected behaviour in Apache Tomcat (version > 10.1.19) > > > where we see that HTTP/1.1 connections are closed whenever a servlet > > > application returns the following status codes: 400, 408, 411, 414, > 500, > > > 503, 501. This causes client applications to rapidly reconnect and > induce > > > high server-side CPU load due to doing TLS handshakes. > > > > > > The 400 and 500 status codes are commonly used in RESTful > microservices to > > > communicate errors. Reviewing RFC 9112 I couldn't find any requirement > for > > > closing connections on these status codes. > > > > > > After testing with Undertow (version 2.3.12), where we didn't see the > same > > > behaviour, we believe that these status codes do not necessitate a new > > > connection. > > > > The Tomcat developers disagree. Connections are closed after these > > status codes to avoid various forms of request smuggling attacks. > > > > > Checking the Tomcat sources makes me believe that the behaviour is > > > hard-coded[1]. I'm reaching out here to re-evaluate the list of status > > > codes and to discuss the possibilities of making the behaviour > configurable. > > > > Making this list of status codes configurable seems reasonable. The > > default can stay as current and if users want to change it then they > > have to accept the associated security risks. > > If it's insecure, then it would still be a valid CVE even if the > configuration is optional. We don't have an "allowSmuggling" attribute > on the connector to relax header or status line parsing, even though > many would have wanted it in the past ... > > Rémy > > > Mark > > > > > > > > > > A colleague of mine reported a bug for this issue: > > > https://bz.apache.org/bugzilla/show_bug.cgi?id=68901 > > > > > > Kind regards, > > > Stefan Ansing > > > > > > [1]: > > > > https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617 > > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat closes connections on unexpected status codes
On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas wrote: > > On 18/04/2024 09:07, Stefan Ansing wrote: > > Hi, > > > > We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19) > > where we see that HTTP/1.1 connections are closed whenever a servlet > > application returns the following status codes: 400, 408, 411, 414, 500, > > 503, 501. This causes client applications to rapidly reconnect and induce > > high server-side CPU load due to doing TLS handshakes. > > > > The 400 and 500 status codes are commonly used in RESTful microservices to > > communicate errors. Reviewing RFC 9112 I couldn't find any requirement for > > closing connections on these status codes. > > > > After testing with Undertow (version 2.3.12), where we didn't see the same > > behaviour, we believe that these status codes do not necessitate a new > > connection. > > The Tomcat developers disagree. Connections are closed after these > status codes to avoid various forms of request smuggling attacks. > > > Checking the Tomcat sources makes me believe that the behaviour is > > hard-coded[1]. I'm reaching out here to re-evaluate the list of status > > codes and to discuss the possibilities of making the behaviour configurable. > > Making this list of status codes configurable seems reasonable. The > default can stay as current and if users want to change it then they > have to accept the associated security risks. If it's insecure, then it would still be a valid CVE even if the configuration is optional. We don't have an "allowSmuggling" attribute on the connector to relax header or status line parsing, even though many would have wanted it in the past ... Rémy > Mark > > > > > > A colleague of mine reported a bug for this issue: > > https://bz.apache.org/bugzilla/show_bug.cgi?id=68901 > > > > Kind regards, > > Stefan Ansing > > > > [1]: > > https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617 > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
On 18/04/2024 09:07, Stefan Ansing wrote: Hi, We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19) where we see that HTTP/1.1 connections are closed whenever a servlet application returns the following status codes: 400, 408, 411, 414, 500, 503, 501. This causes client applications to rapidly reconnect and induce high server-side CPU load due to doing TLS handshakes. The 400 and 500 status codes are commonly used in RESTful microservices to communicate errors. Reviewing RFC 9112 I couldn't find any requirement for closing connections on these status codes. After testing with Undertow (version 2.3.12), where we didn't see the same behaviour, we believe that these status codes do not necessitate a new connection. The Tomcat developers disagree. Connections are closed after these status codes to avoid various forms of request smuggling attacks. Checking the Tomcat sources makes me believe that the behaviour is hard-coded[1]. I'm reaching out here to re-evaluate the list of status codes and to discuss the possibilities of making the behaviour configurable. Making this list of status codes configurable seems reasonable. The default can stay as current and if users want to change it then they have to accept the associated security risks. Mark A colleague of mine reported a bug for this issue: https://bz.apache.org/bugzilla/show_bug.cgi?id=68901 Kind regards, Stefan Ansing [1]: https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org