Hi user group:
just found open-telemetry has fixing for it:
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/tomcat/tomcat-common/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/tomcat/common/TomcatHelper.java#L71
```
static String messageBytesToString(MessageBytes messageBytes) {
// on tomcat 10.1.0 MessageBytes.toString() has a side effect. Calling
it caches the string
// value and changes type of the MessageBytes from T_BYTES to T_STR
which breaks request
// processing in CoyoteAdapter.postParseRequest when it is called on
MessageBytes from
// request.requestURI().
if (messageBytes.getType() == MessageBytes.T_BYTES) {
return messageBytes.getByteChunk().toString();
}
return messageBytes.toString();
}
}
```
so I will update our telemetry-agent first
thanks!
Zhou Rui
On Sat, 18 Mar 2023 at 11:33, Rui <wireles...@gmail.com> wrote:
> Hi Mark
>
> I can't find any Valve setting in the spring boot application(search by
> valve), would you mind sharing more info?
>
> Regarding the mb state and debugging, yes aware that my local intellij
> debugger could change the mb state, so in the the debug lib I call
> mb.getType() only
>
> I got more interesting logs today:
>
> step 1. a stack trace before it hits CoyoteAdapter.java#L677
> <https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677>
>
> ``` I type manually, can't copy it
> mb.toString()
>
> io.opentelemetry.javaagent.instrumentation.tomcat.common.TomcatHttpAttributesGetter.target(TomcatHttpAttributesGetter.java:26)
> ...
> ```
> refer to
>
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/tomcat/tomcat-common/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/tomcat/common/TomcatHttpAttributesGetter.java#L28
>
> so the thing is telemetry-agent adds instrumentation code to call
> mb.toString() before it hits tomcat, which will change mb type(to T_STR)
>
>
> step 2.
> https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
> mb type was changed at step1 so the following processing is wrong.
>
> Questions:
> 1. In mb.toString() it will change the type to (T_STR), is it a right
> approach?(compare with 9.0.70)
> 2. It may not be tomcat bug, but I feel a bit confused. The type will be
> changed to T_STR implicitly when call mb.toString(), it is kind of API
> change will impact 3rd libs such as open-telemetry?
> 3. what is your suggestion to solve this issue?
>
>
> thanks
> Zhou Rui
>
> On Fri, 17 Mar 2023 at 22:41, Mark Thomas <ma...@apache.org> wrote:
>
>> On 17/03/2023 14:27, Rui wrote:
>> > Hi user group:
>> >
>> > I added some debug info to the 9.0.71 baseline
>> >
>> > from the logs it hit this (supposed not inn 9.0.70)
>> >
>> https://github.com/apache/tomcat/blob/9.0.71/java/org/apache/catalina/connector/CoyoteAdapter.java#L677
>>
>> That suggests you are using the Rewrite Valve? Is that correct?
>>
>> If so, that suggests that whatever is going wrong, is going wrong in
>> that Valve.
>>
>> If not, it is likely that your debugging is corrupting the state of the
>> MessageByte instances. You need to be really careful when logging values
>> from them.
>>
>> Mark
>>
>> >
>> > means undecodedURI type is T_CHARS or T_STR, however, decodedURI type is
>> > T_NULL at this time,
>> > so decodedURI.toChars(); actually will recycle its buffer.
>> >
>> > I can reproduce it locally with debug mode, but don't know how
>> reproduce it
>> > in the company EKS cluster
>> >
>> > thanks.
>> > Zhou Rui
>> >
>> >
>> >
>> > On Fri, 17 Mar 2023 at 00:50, Rui <wireles...@gmail.com> wrote:
>> >
>> >> I did some tests with several different commits in 9.0.71, I think my
>> >> issue is caused by this
>> >>
>> >>
>> https://github.com/apache/tomcat/commit/10a1a6d46d952bab4dfde44c3c0de12b0330da79
>> >> the "toBytesSimple" change has not been added to the repo yet, so the
>> >> change in 9.073 doesn't solve the problem.
>> >> Next I will go through these codes, but any clue?
>> >>
>> >> thanks
>> >> Zhou Rui
>> >>
>> >>
>> >> On Tue, 7 Mar 2023 at 00:34, Mark Thomas <ma...@apache.org> wrote:
>> >>
>> >>> On 25/02/2023 17:57, Mark Thomas wrote:
>> >>>>
>> >>>>
>> >>>> On 25/02/2023 15:47, Rui wrote:
>> >>>>> Hi
>> >>>>> recently upgraded tomcat to 9.0.71 from 9.0.70
>> >>>>> but saw 404 in our EKS cluster(with istio installed)
>> >>>>>
>> >>>>> Received [GET /actuator HTTP/1.1
>> >>>>> Host: xxxxx:8079
>> >>>>> User-Agent: kube-probe/1.23+
>> >>>>> Accept: */*
>> >>>>> Connection: close
>> >>>>> Accept-Encoding: gzip
>> >>>>> ]
>> >>>>> Incoming request /health with originalRemoteAddr xxxx
>> >>>>>
>> >>>>> in 9.0.70 I can see the below following message but not in 9.0.71
>> >>>>> o.a.c.authenticator.AuthenticatorBase : Security checking request
>> >>> GET
>> >>>>> /health
>> >>>>>
>> >>>>> seems the processing has stopped somewhere and the pod health check
>> >>>>> didn't
>> >>>>> pass.
>> >>>>>
>> >>>>> I also noticed there was also a 404 issue but don't know if it is
>> >>>>> relevant.
>> >>>>> https://lists.apache.org/thread/gr814rmrlbk9rrqxqjrh4p3x0bfvv1g9
>> >>>>>
>> >>>>> I have tested it locally with curl or jmeter, but can't reproduce
>> >>>>> the problem.
>> >>>>>
>> >>>>> but when I step by step debug the spring app, I found the
>> undecodedURI
>> >>>>> type
>> >>>>> is T_STR
>> >>>>> in CoyoteAdapter.java(in 70 it supposed to be T_BYTES), then
>> >>>>> decodedURI is
>> >>>>> uninitialized and the uri can't find the mapping in internalMap of
>> >>>>> Mapper.java, which will cause 404(guess it thinks the uri is
>> malformed)
>> >>>>>
>> >>>>> MessageBytes decodedURI = req.decodedURI();
>> >>>>>
>> >>>>> if (undecodedURI.getType() == MessageBytes.T_BYTES) {
>> >>>>> // Copy the raw URI to the decodedURI
>> >>>>> decodedURI.duplicate(undecodedURI);
>> >>>>>
>> >>>>>
>> >>>>> 404 example: (when I debug step by step to check unndecodedURI)
>> >>>>>
>> >>>>> curl http://localhost:8080/actuator
>> >>>>>
>> >>>>> <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not
>> >>>>> Found</title><style type="text/css">body
>> >>>>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
>> >>>>> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
>> >>>>> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a
>> >>>>> {color:black;}
>> >>>>> .line
>> >>>>>
>> >>>
>> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
>> >>>>> Status 404 – Not Found</h1></body></html>%
>> >>>>>
>> >>>>>
>> >>>>> However, it runs well without breakpoint! I am quite confused...my
>> >>>>> intellij
>> >>>>> has problem?
>> >>>>> but anyway, summary:
>> >>>>> 9.0.69 and 70 are good in the AWS/EKS(kubernetes) cluster with
>> istio
>> >>>>> 9.0.71 and 72 return 404 when health check by the EKS cluster.
>> >>>>
>> >>>> Looks like an instance of:
>> >>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66488
>> >>>
>> >>> If not that then maybe
>> >>>
>> >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=66512
>> >>>
>> >>> Mark
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >>>
>> >>>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
