> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Friday, October 16, 2020 5:17 AM
> To: users@tomcat.apache.org
> Subject: Re: Weirdest Tomcat Behavior Ever?
>
> On 16/10/2020 10:05, Eric Robinson wrote:
> > Hi Mark --
> >
> > Those are great questions. See answers below.
> >
> >
> >> -----Original Message-----
> >> From: Mark Thomas <ma...@apache.org>
> >> Sent: Friday, October 16, 2020 2:20 AM
> >> To: users@tomcat.apache.org
> >> Subject: Re: Weirdest Tomcat Behavior Ever?
> >>
> >> On 16/10/2020 00:27, Eric Robinson wrote:
> >>
> >> <snip/>
> >>
> >>> The localhost_access log shows a request received and an HTTP 200
> >> response sent, as follows...
> >>>
> >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET
> >>> /app/code.jsp?gizmoid=64438&weekday=5&aptdate=2020-10-
> >> 15&multiResFacFi
> >>>
> >>
> lterId=0&sessionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappproc
> >> e
> >>> ssid=40696&rnd2=0.0715816×tamp=15102020125245.789063
> HTTP/1.0
> >>> ?gizmoid=64438&weekday=5&aptdate=2020-10-
> >> 15&multiResFacFilterId=0&sess
> >>>
> >>
> ionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappprocessid=40696&
> >> rn
> >>> d2=0.0715816×tamp=15102020125245.789063 200
> >>>
> >>> But WireShark shows what really happened. The server received the
> >>> GET
> >> request, and then it sent a FIN to terminate the connection. So if
> >> tomcat sent an HTTP response, it did not make it out the Ethernet card.
> >>>
> >>> Is this the weirdest thing or what? Ideas would sure be appreciated!
> >>
> >> I am assuming there is a typo in your Java version and you are using Java
> >> 8.
> >>
> >
> > Yes, Java 8.
> >
> >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in
> >> less than 6 months). If you aren't already planning to upgrade (I'd
> >> suggest to 9.0.x) then you might want to start thinking about it.
> >>
> >
> > Vendor constraint. It's a canned application published by a national
> software company, and they have not officially approved tomcat 8 for use on
> Linux yet.
> >
> >> I have a few ideas about what might be going on but rather than fire
> >> out random theories I have some questions that might help narrow things
> down.
> >>
> >> 1. If this request was successful, how big is the response?
> >>
> >
> > 1035 bytes.
> >
> >> 2. If this request was successful, how long would it typically take
> >> to complete?
> >>
> >
> > Under 60 ms.
> >
> >> 3. Looking at the Wireshark trace for a failed request, how long
> >> after the last byte of the request is sent by the client does Tomcat send
> the FIN?
> >>
> >
> > Maybe 100 microseconds.
> >
> >> 4. Looking at the Wireshark trace for a failed request, is the
> >> request fully sent (including terminating CRLF etc)?
> >>
> >
> > Yes, the request as seen by the tomcat server is complete and is
> terminated by 0D 0A.
> >
> >> 5. Are there any proxies, firewalls etc between the user agent and
> Tomcat?
> >>
> >
> > User agent -> firewall -> nginx plus -> upstream tomcat servers
> >
> >> 6. What timeouts are configured for the Connector?
> >>
> >
> > Sorry, which connector are you referring to?
> >
> >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS?
> >>
> >
> > HTTP/1.1
> >
> >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere
> >> else?
> >
> > On the nginx proxy and both upstream tomcat servers. (On the user
> > agent, too, but that doesn't help us in this case.)
> >
> > If you would like to see a screen shot showing all 4 captures side-by-size,
> > I
> can send you a secure link. It will verify my answers above. It shows 4
> separate WireShark captures taken simultaneously:
> >
> > (a) the request going from the nginx proxy to tomcat 1
> > (b) tomcat 1 receiving the request and terminating the connection
> > (c) nginx sending the request to tomcat 2
> > (d) tomcat 2 replying to the request (but the reply does not help the user
> because the tomcat server does not recognize the user agent's JSESSIONID
> cookie, so it responds "invalid session."
>
> Hmm.
>
> That rules out most of my ideas.
>
> I'd like to see those screen shots please. Better still would be access to the
> captures themselves (just the relevant connections not the whole thing). I
> believe what you are telling us but long experience tells me it is best to
> double check the original data as well.
>
I'll send you a link to the screen shot first, then I'll package up the
captures and send a link to that in a bit. As the files may contain somewhat
sensitive information, I'll send a secure mail direct to your inbox.
> I have observed something similar ish in the CI systems. In that case it is
> the
> requests that disappear. Client side logging shows the request was made but
> there is no sign of it ever being received by Tomcat. I don't have network
> traces for that (yet) so I'm not sure where the data is going missing.
>
> I am beginning to suspect there is a hard to trigger Tomcat or JVM bug here. I
> think a Tomcat bug is more likely although I have been over the code several
> times and I don't see anything.
>
I'm thinking a bug of some kind, too, but I've been hosting about 1800
instances of tomcat for 15 years and I have never seen this behavior before.
> A few more questions:
>
This is where I will begin to struggle bit.
> Which HTTP connector are you using? BIO, NIO or APR/Native?
>
I believe BIO is the default? server.xml just says...
<Connector port="3016" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
> Is the issue reproducible if you switch to a different connector?
>
In 15 years of using tomcat in production, we've never tried switching the
connector type. (Probably because the app vendor never suggested it.) I did a
little research and I'm beginning to think about the pros/cons.
> How easy is it for you to reproduce this issue?
>
It's not reproducible at will but it happens frequently enough that we don't
have to wait long for it to happen. I have wireshark capturing to disk
continuously and rotating the files at 10 minute intervals to keep them
smallish. Then I just tail the logs and wait.
> How are you linking the request you see in the access log with the request
> you see in Wireshark?
>
Aside from the timestamp of the packets and the timestamp of the tomcat log
messages, each HTTP request also contains a high-resolution timestamp and a
unique random number. That way, even if the same request occurs multiple times
in rapid succession, we can still isolate the exact one that failed.
> How comfortable are you running a patched version of Tomcat (drop class
> files provided by me into $CATALINA_BASE/lib in the right directory structure
> and restart Tomcat)? Just thinking ahead about collecting additional debug
> information.
>
That would be a tricky in our production environment, but the users are getting
desperate enough that we'd be willing to explore that approach.
> Thanks,
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
