Hi Eric, These weird situations are sometimes best looked at by confirming what the OS is seeing from user-space.
Can you run: sudo strace -r -f -e trace=network -p <tomcat pid> You can then log that to a file and correlate and see if the kernel is in fact being asked to send the response. It's very insightful to see what is actually going on between the JVM and Kernel. Paul On Fri, Oct 16, 2020 at 12:16 PM Mark Thomas <ma...@apache.org> wrote: > On 16/10/2020 10:05, Eric Robinson wrote: > > Hi Mark -- > > > > Those are great questions. See answers below. > > > > > >> -----Original Message----- > >> From: Mark Thomas <ma...@apache.org> > >> Sent: Friday, October 16, 2020 2:20 AM > >> To: users@tomcat.apache.org > >> Subject: Re: Weirdest Tomcat Behavior Ever? > >> > >> On 16/10/2020 00:27, Eric Robinson wrote: > >> > >> <snip/> > >> > >>> The localhost_access log shows a request received and an HTTP 200 > >> response sent, as follows... > >>> > >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > >>> /app/code.jsp?gizmoid=64438&weekday=5&aptdate=2020-10- > >> 15&multiResFacFi > >>> > >> lterId=0&sessionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappproc > >> e > >>> ssid=40696&rnd2=0.0715816×tamp=15102020125245.789063 HTTP/1.0 > >>> ?gizmoid=64438&weekday=5&aptdate=2020-10- > >> 15&multiResFacFilterId=0&sess > >>> > >> ionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappprocessid=40696& > >> rn > >>> d2=0.0715816×tamp=15102020125245.789063 200 > >>> > >>> But WireShark shows what really happened. The server received the GET > >> request, and then it sent a FIN to terminate the connection. So if > tomcat sent > >> an HTTP response, it did not make it out the Ethernet card. > >>> > >>> Is this the weirdest thing or what? Ideas would sure be appreciated! > >> > >> I am assuming there is a typo in your Java version and you are using > Java 8. > >> > > > > Yes, Java 8. > > > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less > than 6 > >> months). If you aren't already planning to upgrade (I'd suggest to > 9.0.x) then > >> you might want to start thinking about it. > >> > > > > Vendor constraint. It's a canned application published by a national > software company, and they have not officially approved tomcat 8 for use on > Linux yet. > > > >> I have a few ideas about what might be going on but rather than fire out > >> random theories I have some questions that might help narrow things > down. > >> > >> 1. If this request was successful, how big is the response? > >> > > > > 1035 bytes. > > > >> 2. If this request was successful, how long would it typically take to > >> complete? > >> > > > > Under 60 ms. > > > >> 3. Looking at the Wireshark trace for a failed request, how long after > the last > >> byte of the request is sent by the client does Tomcat send the FIN? > >> > > > > Maybe 100 microseconds. > > > >> 4. Looking at the Wireshark trace for a failed request, is the request > fully sent > >> (including terminating CRLF etc)? > >> > > > > Yes, the request as seen by the tomcat server is complete and is > terminated by 0D 0A. > > > >> 5. Are there any proxies, firewalls etc between the user agent and > Tomcat? > >> > > > > User agent -> firewall -> nginx plus -> upstream tomcat servers > > > >> 6. What timeouts are configured for the Connector? > >> > > > > Sorry, which connector are you referring to? > > > >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? > >> > > > > HTTP/1.1 > > > >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere > >> else? > > > > On the nginx proxy and both upstream tomcat servers. (On the user agent, > too, but that doesn't help us in this case.) > > > > If you would like to see a screen shot showing all 4 captures > side-by-size, I can send you a secure link. It will verify my answers > above. It shows 4 separate WireShark captures taken simultaneously: > > > > (a) the request going from the nginx proxy to tomcat 1 > > (b) tomcat 1 receiving the request and terminating the connection > > (c) nginx sending the request to tomcat 2 > > (d) tomcat 2 replying to the request (but the reply does not help the > user because the tomcat server does not recognize the user agent's > JSESSIONID cookie, so it responds "invalid session." > > Hmm. > > That rules out most of my ideas. > > I'd like to see those screen shots please. Better still would be access > to the captures themselves (just the relevant connections not the whole > thing). I believe what you are telling us but long experience tells me > it is best to double check the original data as well. > > I have observed something similar ish in the CI systems. In that case it > is the requests that disappear. Client side logging shows the request > was made but there is no sign of it ever being received by Tomcat. I > don't have network traces for that (yet) so I'm not sure where the data > is going missing. > > I am beginning to suspect there is a hard to trigger Tomcat or JVM bug > here. I think a Tomcat bug is more likely although I have been over the > code several times and I don't see anything. > > A few more questions: > > Which HTTP connector are you using? BIO, NIO or APR/Native? > > Is the issue reproducible if you switch to a different connector? > > How easy is it for you to reproduce this issue? > > How are you linking the request you see in the access log with the > request you see in Wireshark? > > How comfortable are you running a patched version of Tomcat (drop class > files provided by me into $CATALINA_BASE/lib in the right directory > structure and restart Tomcat)? Just thinking ahead about collecting > additional debug information. > > Thanks, > > Mark > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
