________________________________ From: i...@flyingfischer.ch <i...@flyingfischer.ch> Sent: Monday, December 5, 2016 8:46 AM To: Struts Developers List Subject: Re: Valid characters in http requets: Tomcat 8.38 -> 8.39
Martin Maybe you should send one to the user-list, which is the offical disccusion list. I will gladly assist and comment. If I do repost to MG>snip MG>if this is a genuine bug that needs to be on someones radar then we should put it there Please be aware that the current change in behaviour between 8.0.38 and 8.0.39 is related to a security fix: https://tomcat.apache.org/security-8.html#Fixed_in_Apache_Tomcat_8.0.39 The question then is: is rejecting the character '|' in a query part for a request target the appropriate measure to take against it? MG>his argument for 400 Bad Request is to categorise | as whitespace MG>way back in 02 when shapiro was lead whitespace was only #x9 tab, #xA linefeed, #xD carriage-return MG>https://books.google.com/books?id=127iOj4MDzMC&pg=PA323&lpg=PA323&dq=RFC+and+%22whitespace+definition%22&source=bl&ots=RoMy6Del18&sig=-f8anhUqTj0IfbycvLhnw6gjGC8&hl=en&sa=X&ved=0ahUKEwjkzsTupd3QAhUU8GMKHYRTCjoQ6AEIKDAC#v=onepage&q=whitespace&f=false <https://books.google.com/books?id=127iOj4MDzMC&pg=PA323&lpg=PA323&dq=RFC+and+%22whitespace+definition%22&source=bl&ots=RoMy6Del18&sig=-f8anhUqTj0IfbycvLhnw6gjGC8&hl=en&sa=X&ved=0ahUKEwjkzsTupd3QAhUU8GMKHYRTCjoQ6AEIKDAC#v=onepage&q=whitespace&f=false> MG>later in 2005 ...rfc-3986 whitespace was defined as (spaces, line-breaks, tabs, etc.).. but what exactly is etc? https://www.ietf.org/rfc/rfc3986.txt RFC 3986 - Internet Engineering Task Force<https://www.ietf.org/rfc/rfc3986.txt> www.ietf.org ... RFC 3986 URI Generic Syntax January 2005 D dec-octet 20 dereference 9 dot-segments 23 F fragment 16, 24 G gen-delims 13 generic syntax 6 H ... MG>so as of 2005 it seems we are "safe" that | is not classified whitespace.. MG>https://tools.ietf.org/html/rfc7230#section-3.2.4 <https://tools.ietf.org/html/rfc7230#section-3.2.4>MG>but IS the character | a whitespace char?..lets use rfc2730 as our reference here is the text on Whitespace v Delimiter 3.2.6<https://tools.ietf.org/html/rfc7230#section-3.2.6>. Field Value Components Most HTTP header field values are defined using common syntax components (token, quoted-string, and comment) separated by whitespace or specific delimiting characters. Delimiters are chosen from the set of US-ASCII visual characters not allowed in a token (DQUOTE and "(),/:;<=>?@[\]{}"). token = 1*tchar tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA ; any VCHAR, except delimiters MG>so our argument is "|" is a recognised delimiter and not a whitespace MG>also the reason for him classifying "|" as whitespace would also lead him to reject "!" (which is struts bang character) The choices to be rejected seem to be quite random: MG>agree // Combination of multiple rules from RFC7230 and RFC 3986. Must be // ASCII, no controls plus a few additional characters excluded http://svn.apache.org/viewvc/tomcat/tc8.0.x/trunk/java/org/apache/tomcat/util/http/parser/HttpParser.java?view=markup&pathrev=1767653 MG>Request by Markus and Martin MG>please revert security fix specifically for character | to allow passing of | character by HttpParser.java The general pupose of this parser is described as follows: "The purpose of this parser is to let the parser worry about the edge cases. It provides tolerant (where safe to do so) parsing of HTTP header values assuming that wrapped header lines have already been unwrapped. (The Tomcat header processing code does the unwrapping.)" Best regards Markus You may also reach me off-list, since we are on the struts-list here: i...@flyingfischer.ch Am 05.12.2016 um 14:09 schrieb Martin Gainty: > perhaps we're dealing with a powerful personality that just won 30 of the > states in the US > > > would suggest send a pretty please with sugar on top bug request to tomcat > jira and post to tomcat users > > > https://bz.apache.org/bugzilla/enter_bug.cgi > > > if he unilaterally rejects this obvious bug without consulting his fellow > tomcat committee members we'll need to talk to the "rebel alliance" who has > forked what they label as "commercial tomcat"..the rebels tout their fork as > "hardened tomcat that works for corporations as well as the academic > community" > > > please pingback when you have an answer (or lack of same) > > > *gruss* > > Martin > ______________________________________________ > > > > ________________________________ > From: i...@flyingfischer.ch <i...@flyingfischer.ch> > Sent: Monday, December 5, 2016 3:33 AM > To: Struts Developers List > Subject: Re: Valid characters in http requets: Tomcat 8.38 -> 8.39 > > Dear Martin > > I fear Tomcat will not fix the issue. Mark Thomas states: > > "If a request contains and unencoded '|' in the request-target, the > correct way to deal with it is to return a 400." > > While this may be true, and when ever you are able to fix the requests > made to Tomcat, you should do it. However, if you are in a situation > like me, where this is not feasible, I see two options: > > 1. try to post again to the user-list in Tomcat, to raise awareness of > the issue > > 2. patch Tomcat > > While I do not like to do this, patching Tomcat is very easy: > > svn checkout http://svn.apache.org/repos/asf/tomcat/tc8.0.x/trunk/ asf - Revision 1769242: /tomcat/tc8.0.x/trunk<http://svn.apache.org/repos/asf/tomcat/tc8.0.x/trunk/> svn.apache.org asf - Revision 1769242: /tomcat/tc8.0.x/trunk...gitignore; BUILDING.txt; KEYS; LICENSE; MERGE.txt; NOTICE; RELEASE-NOTES; RUNNING.txt; TOMCAT-NEXT.txt; bin/ build ... > asf - Revision 1769242: > /tomcat/tc8.0.x/trunk<http://svn.apache.org/repos/asf/tomcat/tc8.0.x/trunk/> > svn.apache.org > asf - Revision 1769242: /tomcat/tc8.0.x/trunk...gitignore; BUILDING.txt; > KEYS; LICENSE; MERGE.txt; NOTICE; RELEASE-NOTES; RUNNING.txt; > TOMCAT-NEXT.txt; bin/ build ... > > > > > Take a look at > > java/org/apache/tomcat/util/http/parser/HttpParser.java > > cd {tomcat}/trunk/ > ant > > and you'll find the result in {tomcat}/trunk/output/build/ > Markus > > >> Dear Martin >> >> I fear Tomcat will not fix the issue. Mark Thomas states: >> >> "If a request contains and unencoded '|' in the request-target, the >> correct way to deal with it is to return a 400." >> >> While this may be true, and when ever you are able to fix the requests >> made to Tomcat, you should do it. However, if you are in a situation >> like me, where this is not feasible, I see two options: >> >> * try to post again to the user-list in Tomcat, to raise awareness of >> the issue >> * patch Tomcat >> >> While I do not like to do this, patching Tomcat is very easy: >> >> svn checkout http://svn.apache.org/repos/asf/tomcat/tc8.0.x/trunk/ asf - Revision 1769242: /tomcat/tc8.0.x/trunk<http://svn.apache.org/repos/asf/tomcat/tc8.0.x/trunk/> svn.apache.org asf - Revision 1769242: /tomcat/tc8.0.x/trunk...gitignore; BUILDING.txt; KEYS; LICENSE; MERGE.txt; NOTICE; RELEASE-NOTES; RUNNING.txt; TOMCAT-NEXT.txt; bin/ build ... > asf - Revision 1769242: > /tomcat/tc8.0.x/trunk<http://svn.apache.org/repos/asf/tomcat/tc8.0.x/trunk/> > svn.apache.org > asf - Revision 1769242: /tomcat/tc8.0.x/trunk...gitignore; BUILDING.txt; > KEYS; LICENSE; MERGE.txt; NOTICE; RELEASE-NOTES; RUNNING.txt; > TOMCAT-NEXT.txt; bin/ build ... > > > >> >> Take a look at >> >> java/org/apache/tomcat/util/http/parser/HttpParser.java >> >> cd {tomcat}/trunk/ >> ant >> >> and you'll find the result in {tomcat}/trunk/output/build/ >> >> Markus >> >> Am 04.12.2016 um 02:30 schrieb Martin Gainty: >>> Markus >>> >>> I have same problem and had to revert TC 8.38 ..please pingback when tomcat >>> fixed this problem >>> >>> >>> *gruss* >>> >>> Martin >>> ____________ >>> >>> >>> >>> ________________________________ >>> From: i...@flyingfischer.ch <i...@flyingfischer.ch> >>> Sent: Saturday, December 3, 2016 8:18 AM >>> To: Struts Developers List >>> Subject: Re: Valid characters in http requets: Tomcat 8.38 -> 8.39 >>> >>> Sorry! Wrong mailing list... >>> >>> Markus >>> >>> Am 03.12.2016 um 13:56 schrieb Lukasz Lenart: >>>> Is it related to Apache Struts? >>>> >>>> >>>> Cheers >>>> Lukasz >>>> >>>> 2016-12-03 12:47 GMT+01:00 i...@flyingfischer.ch <i...@flyingfischer.ch>: >>>>> Between Tomcat 8.38 und 8.39 there seems to be a change in handling URL >>>>> parameters: >>>>> >>>>> ¶mxy=1|2 >>>>> >>>>> This will cause Tomcat to return a 400 error since 8.39. It is the >>>>> character >>>>> "|" that causes the new behaviour. I suspect these changes: >>>>> >>>>> https://github.com/apache/tomcat/commit/516bda676ac8d0284da3e0295a7df70391315360 [https://avatars3.githubusercontent.com/u/4690029?v=3&s=200]<https://github.com/apache/tomcat/commit/516bda676ac8d0284da3e0295a7df70391315360> Add additional checks for valid characters to the HTTP request line · apache/tomcat@516bda6<https://github.com/apache/tomcat/commit/516bda676ac8d0284da3e0295a7df70391315360> github.com parsing so invalid request lines are rejected sooner. git-svn-id: https://svn.apache.org/repos/asf/tomcat/trunk@1767641 13f79535-47bb-0310-9956-ffa450edef68 > [https://avatars3.githubusercontent.com/u/4690029?v=3&s=200]<https://github.com/apache/tomcat/commit/516bda676ac8d0284da3e0295a7df70391315360> > > Add additional checks for valid characters to the HTTP request line · > apache/tomcat@516bda6<https://github.com/apache/tomcat/commit/516bda676ac8d0284da3e0295a7df70391315360> > github.com > parsing so invalid request lines are rejected sooner. git-svn-id: > https://svn.apache.org/repos/asf/tomcat/trunk@1767641 > 13f79535-47bb-0310-9956-ffa450edef68 > > > >>> [https://avatars3.githubusercontent.com/u/4690029?v=3&s=200]<https://github.com/apache/tomcat/commit/516bda676ac8d0284da3e0295a7df70391315360> >>> >>> Add additional checks for valid characters to the HTTP request line · >>> apache/tomcat@516bda6<https://github.com/apache/tomcat/commit/516bda676ac8d0284da3e0295a7df70391315360> >>> github.com >>> parsing so invalid request lines are rejected sooner. git-svn-id: >>> https://svn.apache.org/repos/asf/tomcat/trunk@1767641 >>> 13f79535-47bb-0310-9956-ffa450edef68 >>> >>> >>> >>>>> First thing to know: >>>>> >>>>> Is this intended? >>>>> >>>>> Second: >>>>> >>>>> Anyway to restore the previous behaviour of 8.38 with a config option. >>>>> >>>>> Thanks for considering! >>>>> >>>>> Best regards >>>>> Markus >>>>> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@struts.apache.org For additional commands, e-mail: dev-h...@struts.apache.org
