Hi Mark I can't find any Valve setting in the spring boot application(search by valve), would you mind sharing more info?
Regarding the mb state and debugging, yes aware that my local intellij debugger could change the mb state, so in the the debug lib I call mb.getType() only I got more interesting logs today: step 1. a stack trace before it hits CoyoteAdapter.java#L677 <https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677> ``` I type manually, can't copy it mb.toString() io.opentelemetry.javaagent.instrumentation.tomcat.common.TomcatHttpAttributesGetter.target(TomcatHttpAttributesGetter.java:26) ... ``` refer to https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/tomcat/tomcat-common/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/tomcat/common/TomcatHttpAttributesGetter.java#L28 so the thing is telemetry-agent adds instrumentation code to call mb.toString() before it hits tomcat, which will change mb type(to T_STR) step 2. https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677 mb type was changed at step1 so the following processing is wrong. Questions: 1. In mb.toString() it will change the type to (T_STR), is it a right approach?(compare with 9.0.70) 2. It may not be tomcat bug, but I feel a bit confused. The type will be changed to T_STR implicitly when call mb.toString(), it is kind of API change will impact 3rd libs such as open-telemetry? 3. what is your suggestion to solve this issue? thanks Zhou Rui On Fri, 17 Mar 2023 at 22:41, Mark Thomas <ma...@apache.org> wrote: > On 17/03/2023 14:27, Rui wrote: > > Hi user group: > > > > I added some debug info to the 9.0.71 baseline > > > > from the logs it hit this (supposed not inn 9.0.70) > > > https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677 > > That suggests you are using the Rewrite Valve? Is that correct? > > If so, that suggests that whatever is going wrong, is going wrong in > that Valve. > > If not, it is likely that your debugging is corrupting the state of the > MessageByte instances. You need to be really careful when logging values > from them. > > Mark > > > > > means undecodedURI type is T_CHARS or T_STR, however, decodedURI type is > > T_NULL at this time, > > so decodedURI.toChars(); actually will recycle its buffer. > > > > I can reproduce it locally with debug mode, but don't know how reproduce > it > > in the company EKS cluster > > > > thanks. > > Zhou Rui > > > > > > > > On Fri, 17 Mar 2023 at 00:50, Rui <wireles...@gmail.com> wrote: > > > >> I did some tests with several different commits in 9.0.71, I think my > >> issue is caused by this > >> > >> > https://github.com/apache/tomcat/commit/10a1a6d46d952bab4dfde44c3c0de12b0330da79 > >> the "toBytesSimple" change has not been added to the repo yet, so the > >> change in 9.073 doesn't solve the problem. > >> Next I will go through these codes, but any clue? > >> > >> thanks > >> Zhou Rui > >> > >> > >> On Tue, 7 Mar 2023 at 00:34, Mark Thomas <ma...@apache.org> wrote: > >> > >>> On 25/02/2023 17:57, Mark Thomas wrote: > >>>> > >>>> > >>>> On 25/02/2023 15:47, Rui wrote: > >>>>> Hi > >>>>> recently upgraded tomcat to 9.0.71 from 9.0.70 > >>>>> but saw 404 in our EKS cluster(with istio installed) > >>>>> > >>>>> Received [GET /actuator HTTP/1.1 > >>>>> Host: xxxxx:8079 > >>>>> User-Agent: kube-probe/1.23+ > >>>>> Accept: */* > >>>>> Connection: close > >>>>> Accept-Encoding: gzip > >>>>> ] > >>>>> Incoming request /health with originalRemoteAddr xxxx > >>>>> > >>>>> in 9.0.70 I can see the below following message but not in 9.0.71 > >>>>> o.a.c.authenticator.AuthenticatorBase : Security checking request > >>> GET > >>>>> /health > >>>>> > >>>>> seems the processing has stopped somewhere and the pod health check > >>>>> didn't > >>>>> pass. > >>>>> > >>>>> I also noticed there was also a 404 issue but don't know if it is > >>>>> relevant. > >>>>> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9 > >>>>> > >>>>> I have tested it locally with curl or jmeter, but can't reproduce > >>>>> the problem. > >>>>> > >>>>> but when I step by step debug the spring app, I found the > undecodedURI > >>>>> type > >>>>> is T_STR > >>>>> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then > >>>>> decodedURI is > >>>>> uninitialized and the uri can't find the mapping in internalMap of > >>>>> Mapper.java, which will cause 404(guess it thinks the uri is > malformed) > >>>>> > >>>>> MessageBytes decodedURI = req.decodedURI(); > >>>>> > >>>>> if (undecodedURI.getType() == MessageBytes.T_BYTES) { > >>>>> // Copy the raw URI to the decodedURI > >>>>> decodedURI.duplicate(undecodedURI); > >>>>> > >>>>> > >>>>> 404 example: (when I debug step by step to check unndecodedURI) > >>>>> > >>>>> curl http://localhost:8080/actuator > >>>>> > >>>>> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not > >>>>> Found</title><style type="text/css">body > >>>>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b > >>>>> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 > >>>>> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a > >>>>> {color:black;} > >>>>> .line > >>>>> > >>> > {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP > >>>>> Status 404 – Not Found</h1></body></html>% > >>>>> > >>>>> > >>>>> However, it runs well without breakpoint! I am quite confused...my > >>>>> intellij > >>>>> has problem? > >>>>> but anyway, summary: > >>>>> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with > istio > >>>>> 9.0.71 and 72 return 404 when health check by the EKS cluster. > >>>> > >>>> Looks like an instance of: > >>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66488 > >>> > >>> If not that then maybe > >>> > >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66512 > >>> > >>> Mark > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>> > >>> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
