Embedded vs Standalone Tomcat
My apologies in advance if this has been already discussed in the group. I am looking for experiences of the community, any nitpicks, etc. Currently we are using standalone Tomcat version (9.x) to host a web applications which are essentially hosting REST APIs. We plan to move to micro services model where each web application (logical unit of related APIs) will be hosted as a different process. With this mind, I wanted to evaluate the embedded version of Tomcat. 1. Is there any guidance around when to use what? 2. Are there any feature disparities between standalone Tomcat and embedded version? We have fairly extended Tomcat classes for use cases like automatic certificate renewal, not having to mention is plain-text password in server.xml for key stores, hooking in custom security provider (for FIPS) for Tomcat JVM. etc. So we ideally wouldn't want to lose any of such capabilities when we move to the embedded version. 3. Currently, we have written a Windows SCM service wrapper and manage our Tomcat JVM via that (on Windows). Would there be any issue with embedded Tomcat if we want to have SCM service wrapper? 4. How is the upgrade process different for embedded version? Currently, we ship Tomcat with our product and have a well-defined process to upgrade Tomcat even with older versions of our product deployed in customers' environment. 5. Externalizing the configuration (like we can do that via web.xml, server.xml, context.xml) - any issues, limitations here? 6. Any differences with resource usages (e.g. heap)? Any help is greatly appreciated. Thanks, Amit
RE: Weirdest Tomcat Behavior Ever?
> -Original Message- > From: Mark Thomas > Sent: Friday, October 16, 2020 8:02 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 16/10/2020 12:37, Eric Robinson wrote: > >> From: Mark Thomas > > > > >> I'd like to see those screen shots please. Better still would be > >> access to the captures themselves (just the relevant connections not > >> the whole thing). I believe what you are telling us but long > >> experience tells me it is best to double check the original data as well. > >> > > > > I'll send you a link to the screen shot first, then I'll package up the > > captures > and send a link to that in a bit. As the files may contain somewhat sensitive > information, I'll send a secure mail direct to your inbox. > > Thanks. The screenshots didn't shed any light on this so far. > > >> I have observed something similar ish in the CI systems. In that case > >> it is the requests that disappear. Client side logging shows the > >> request was made but there is no sign of it ever being received by > >> Tomcat. I don't have network traces for that (yet) so I'm not sure where > the data is going missing. > >> > >> I am beginning to suspect there is a hard to trigger Tomcat or JVM > >> bug here. I think a Tomcat bug is more likely although I have been > >> over the code several times and I don't see anything. > >> > > > > I'm thinking a bug of some kind, too, but I've been hosting about 1800 > instances of tomcat for 15 years and I have never seen this behavior before. > > > >> A few more questions: > >> > > > > This is where I will begin to struggle bit. > > > >> Which HTTP connector are you using? BIO, NIO or APR/Native? > >> > > > > I believe BIO is the default? server.xml just says... > > > > >connectionTimeout="2" > >redirectPort="8443" /> > > That will be BIO or APR/Native depending on whether you have Tomcat > Native installed. If you look at the logs for when Tomcat starts you should > see > something like: > > INFO: Initializing ProtocolHandler ["http-bio-3016"] or > INFO: Initializing ProtocolHandler ["http-apr-3016"] > > What do you see between the square brackets? ["http-bio-3016"] > > >> Is the issue reproducible if you switch to a different connector? > >> > > > > In 15 years of using tomcat in production, we've never tried switching the > connector type. (Probably because the app vendor never suggested it.) I did > a little research and I'm beginning to think about the pros/cons. > > If you wanted to try this, I'd recommend: > > protocol="org.apache.coyote.http11.Http11NioProtocol" > We're in the middle of a production day so I want to avoid restarting tomcat if I can, but I'll plan to change that tonight. > >> How easy is it for you to reproduce this issue? > >> > > > > It's not reproducible at will but it happens frequently enough that we don't > have to wait long for it to happen. I have wireshark capturing to disk > continuously and rotating the files at 10 minute intervals to keep them > smallish. Then I just tail the logs and wait. > > Ack. > > >> How are you linking the request you see in the access log with the > >> request you see in Wireshark? > > > > Aside from the timestamp of the packets and the timestamp of the tomcat > log messages, each HTTP request also contains a high-resolution timestamp > and a unique random number. That way, even if the same request occurs > multiple times in rapid succession, we can still isolate the exact one that > failed. > > Excellent. > > >> How comfortable are you running a patched version of Tomcat (drop > >> class files provided by me into $CATALINA_BASE/lib in the right > >> directory structure and restart Tomcat)? Just thinking ahead about > >> collecting additional debug information. > > > > That would be a tricky in our production environment, but the users are > getting desperate enough that we'd be willing to explore that approach. > > Understood. > > Some other questions that have come to mind: > > - Has this app always had this problem? > No, it's been running fine in this environment since October 2018. > - If not, when did it start and what changed at that point (JVM version, > Tomcat version etc) > This is a new thing in the past month or so, but we can't think of what might have changed. There are 2 Linux tomcat servers, each running 17 instances of on different ports. The symptom seems to appear in many of the instances intermittently, but for 2 of them it is painfully frequent. The users are getting kicked out multiple times per day. The tomcat and java versions have not changed since at least July 2019. > - I notice the the requests in the screenshots aren't using HTTP keep-alive. > Do you know why? > That's a good question. I checked the captures from the client side and they do have the HTTP keep-alive header. It must be something related to the proxy. > - How heavily loaded is the system (typical requests per second etc)? >
Re: Virtual event focussed on Tomcat Security
On Thu, Oct 15, 2020 at 2:01 PM Mark Thomas wrote: > On 29/09/2020 12:25, Mark Thomas wrote: > > Hi all, > > > > We (the Tomcat community) have some funding from Google to help us > > improve Tomcat security. Our original plan was to use the funding to > > support an in-person security focussed hackathon. As you would expect, > > those plans are on hold for now. We would, therefore, like to explore > > the possibility of doing something virtually. > > > > The purpose of this email is to gather input from the community about > > what such an event should look like. With that input we can put together > > a plan for the event. So, over to you. What would your ideal virtual > > event focussed on Tomcat Security look like? > > Summarising the suggestions so far: > - application security / OWASP > - making HTTP requests *from* Tomcat > - SSO / SAML / OpenIDConnect > > The first two are more application security focussed and would not have > to be Tomcat specific. > > The third is more likely to Tomcat specific depending on the extent to > which the SSO mechanism ties into Tomcat's internals. > > All the suggestions so far have been for conference like presentations > (if I am reading them correctly). > > Other possibilities: > - hackathon to implement (with support from committers) new security > features (no idea what these might be - suggestions welcome) > > - hackathon to run $tool_of_choice against Tomcat code base, review the > results and fix (with committer support) those that need fixing. > Suggestions as to tools to use welcome* > > Anything else you'd like to suggest that is related to Tomcat and security. > > There hasn't been any thought given to timing yet. > > Mark > > > > * I'll note that over the years most if not all of the major static > analysis tools have been run against the Tomcat code base and the > results have been very heavy on the false positives. Most of the work is > likely to be separating the few useful results from a lot of noise. > > Has a "when" been decided yet? Thanks, Bob
Re: Weirdest Tomcat Behavior Ever?
On 16/10/2020 12:37, Eric Robinson wrote: >> From: Mark Thomas >> I'd like to see those screen shots please. Better still would be access to >> the >> captures themselves (just the relevant connections not the whole thing). I >> believe what you are telling us but long experience tells me it is best to >> double check the original data as well. >> > > I'll send you a link to the screen shot first, then I'll package up the > captures and send a link to that in a bit. As the files may contain somewhat > sensitive information, I'll send a secure mail direct to your inbox. Thanks. The screenshots didn't shed any light on this so far. >> I have observed something similar ish in the CI systems. In that case it is >> the >> requests that disappear. Client side logging shows the request was made but >> there is no sign of it ever being received by Tomcat. I don't have network >> traces for that (yet) so I'm not sure where the data is going missing. >> >> I am beginning to suspect there is a hard to trigger Tomcat or JVM bug here. >> I >> think a Tomcat bug is more likely although I have been over the code several >> times and I don't see anything. >> > > I'm thinking a bug of some kind, too, but I've been hosting about 1800 > instances of tomcat for 15 years and I have never seen this behavior before. > >> A few more questions: >> > > This is where I will begin to struggle bit. > >> Which HTTP connector are you using? BIO, NIO or APR/Native? >> > > I believe BIO is the default? server.xml just says... > > connectionTimeout="2" >redirectPort="8443" /> That will be BIO or APR/Native depending on whether you have Tomcat Native installed. If you look at the logs for when Tomcat starts you should see something like: INFO: Initializing ProtocolHandler ["http-bio-3016"] or INFO: Initializing ProtocolHandler ["http-apr-3016"] What do you see between the square brackets? >> Is the issue reproducible if you switch to a different connector? >> > > In 15 years of using tomcat in production, we've never tried switching the > connector type. (Probably because the app vendor never suggested it.) I did a > little research and I'm beginning to think about the pros/cons. If you wanted to try this, I'd recommend: protocol="org.apache.coyote.http11.Http11NioProtocol" >> How easy is it for you to reproduce this issue? >> > > It's not reproducible at will but it happens frequently enough that we don't > have to wait long for it to happen. I have wireshark capturing to disk > continuously and rotating the files at 10 minute intervals to keep them > smallish. Then I just tail the logs and wait. Ack. >> How are you linking the request you see in the access log with the request >> you see in Wireshark? > > Aside from the timestamp of the packets and the timestamp of the tomcat log > messages, each HTTP request also contains a high-resolution timestamp and a > unique random number. That way, even if the same request occurs multiple > times in rapid succession, we can still isolate the exact one that failed. Excellent. >> How comfortable are you running a patched version of Tomcat (drop class >> files provided by me into $CATALINA_BASE/lib in the right directory structure >> and restart Tomcat)? Just thinking ahead about collecting additional debug >> information. > > That would be a tricky in our production environment, but the users are > getting desperate enough that we'd be willing to explore that approach. Understood. Some other questions that have come to mind: - Has this app always had this problem? - If not, when did it start and what changed at that point (JVM version, Tomcat version etc) - I notice the the requests in the screenshots aren't using HTTP keep-alive. Do you know why? - How heavily loaded is the system (typical requests per second etc)? - I'm assuming there is nothing in the log files when the failed request happens. Is this correct? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: GraalVM native image for embedded Tomcat
On Fri, Oct 16, 2020 at 3:43 PM Martin Grigorov wrote: > Hi, > > I am trying to create a GraalVM native image for an application that uses > tomcat-embed-core but it fails with the following error: > > [INFO] --- native-image-maven-plugin:20.2.0:native-image (default) @ > tomcat-embedded --- > [INFO] ImageClasspath Entry: > org.apache.tomcat.embed:tomcat-embed-core:jar:9.0.39:compile > (file:///home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar) > [INFO] ImageClasspath Entry: > org.apache.tomcat:tomcat-annotations-api:jar:9.0.39:compile > (file:///home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar) > [INFO] ImageClasspath Entry: > info.mgsolutions:tomcat-embedded:jar:1.0-SNAPSHOT > (file:///home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar) > [INFO] Executing: > /home/martin/devel/graalvm-ce-java11-20.2.0/bin/native-image -cp > /home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar:/home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar:/home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar > -H:Class=info.mgsolutions.tomcat.TomcatEmbedded > [info.mgsolutions.tomcat.tomcatembedded:568255]classlist: 2,047.64 > ms, 0.96 GB > [info.mgsolutions.tomcat.tomcatembedded:568255](cap): 852.60 > ms, 0.96 GB > [info.mgsolutions.tomcat.tomcatembedded:568255]setup: 2,323.40 > ms, 0.96 GB > Fatal error:java.util.MissingResourceException: Can't find bundle for base > name org.apache.tomcat.util.descriptor.tld.LocalStrings, locale en_US > at > java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055) > at > java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689) > at > java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593) > at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284) > at > com.oracle.svm.util.ModuleSupport.getResourceBundleFallback(ModuleSupport.java:81) > at > com.oracle.svm.util.ModuleSupport.getResourceBundle(ModuleSupport.java:62) > at > com.oracle.svm.core.jdk.LocalizationFeature.addBundleToCache(LocalizationFeature.java:266) > at > com.oracle.svm.hosted.ResourcesFeature$ResourcesRegistryImpl.addResourceBundles(ResourcesFeature.java:92) > at > com.oracle.svm.core.configure.ResourceConfigurationParser.parseEntry(ResourceConfigurationParser.java:90) > at > com.oracle.svm.core.configure.ResourceConfigurationParser.parseTopLevelObject(ResourceConfigurationParser.java:71) > at > com.oracle.svm.core.configure.ResourceConfigurationParser.parseAndRegister(ResourceConfigurationParser.java:47) > at > com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:132) > at > com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$3(ConfigurationParserUtils.java:119) > at > java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:212) > at > com.oracle.svm.hosted.config.ConfigurationParserUtils$1.tryAdvance(ConfigurationParserUtils.java:108) > at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:326) > at > java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) > at > java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274) > at > java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) > at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) > at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) > at > java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) > at > java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) > at > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) > at > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) > at > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) > at > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:491) > at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:449) > at > com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:125) > at > com.oracle.svm.hosted.ResourcesFeature.beforeAnalysis(ResourcesFeature.java:105) > at > com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$7(NativeImageGenerator.java:693) > at > com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:70) > at >
GraalVM native image for embedded Tomcat
Hi, I am trying to create a GraalVM native image for an application that uses tomcat-embed-core but it fails with the following error: [INFO] --- native-image-maven-plugin:20.2.0:native-image (default) @ tomcat-embedded --- [INFO] ImageClasspath Entry: org.apache.tomcat.embed:tomcat-embed-core:jar:9.0.39:compile (file:///home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar) [INFO] ImageClasspath Entry: org.apache.tomcat:tomcat-annotations-api:jar:9.0.39:compile (file:///home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar) [INFO] ImageClasspath Entry: info.mgsolutions:tomcat-embedded:jar:1.0-SNAPSHOT (file:///home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar) [INFO] Executing: /home/martin/devel/graalvm-ce-java11-20.2.0/bin/native-image -cp /home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar:/home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar:/home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar -H:Class=info.mgsolutions.tomcat.TomcatEmbedded [info.mgsolutions.tomcat.tomcatembedded:568255]classlist: 2,047.64 ms, 0.96 GB [info.mgsolutions.tomcat.tomcatembedded:568255](cap): 852.60 ms, 0.96 GB [info.mgsolutions.tomcat.tomcatembedded:568255]setup: 2,323.40 ms, 0.96 GB Fatal error:java.util.MissingResourceException: Can't find bundle for base name org.apache.tomcat.util.descriptor.tld.LocalStrings, locale en_US at java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055) at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689) at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593) at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284) at com.oracle.svm.util.ModuleSupport.getResourceBundleFallback(ModuleSupport.java:81) at com.oracle.svm.util.ModuleSupport.getResourceBundle(ModuleSupport.java:62) at com.oracle.svm.core.jdk.LocalizationFeature.addBundleToCache(LocalizationFeature.java:266) at com.oracle.svm.hosted.ResourcesFeature$ResourcesRegistryImpl.addResourceBundles(ResourcesFeature.java:92) at com.oracle.svm.core.configure.ResourceConfigurationParser.parseEntry(ResourceConfigurationParser.java:90) at com.oracle.svm.core.configure.ResourceConfigurationParser.parseTopLevelObject(ResourceConfigurationParser.java:71) at com.oracle.svm.core.configure.ResourceConfigurationParser.parseAndRegister(ResourceConfigurationParser.java:47) at com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:132) at com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$3(ConfigurationParserUtils.java:119) at java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:212) at com.oracle.svm.hosted.config.ConfigurationParserUtils$1.tryAdvance(ConfigurationParserUtils.java:108) at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:326) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:491) at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:449) at com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:125) at com.oracle.svm.hosted.ResourcesFeature.beforeAnalysis(ResourcesFeature.java:105) at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$7(NativeImageGenerator.java:693) at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:70) at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:693) at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:555) at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:468)
Re: Weirdest Tomcat Behavior Ever?
This is a bit of a long shot, but are you running on VMWare ESX by chance? A few years ago, I had a problem where Tomcat would send data down the socket, but about half the message wouldn't actually arrive at the client. On Fri, Oct 16, 2020 at 5:05 AM Eric Robinson wrote: > Hi Mark -- > > Those are great questions. See answers below. > > > > -Original Message- > > From: Mark Thomas > > Sent: Friday, October 16, 2020 2:20 AM > > To: users@tomcat.apache.org > > Subject: Re: Weirdest Tomcat Behavior Ever? > > > > On 16/10/2020 00:27, Eric Robinson wrote: > > > > > > > > > The localhost_access log shows a request received and an HTTP 200 > > response sent, as follows... > > > > > > 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > > > /app/code.jsp?gizmoid=64438=5=2020-10- > > 15 > > > > > lterId=0=0=71340=321072 > > e > > > ssid=40696=0.0715816=15102020125245.789063 HTTP/1.0 > > > ?gizmoid=64438=5=2020-10- > > 15=0 > > > > > ionDID=0=71340=321072=40696& > > rn > > > d2=0.0715816=15102020125245.789063 200 > > > > > > But WireShark shows what really happened. The server received the GET > > request, and then it sent a FIN to terminate the connection. So if > tomcat sent > > an HTTP response, it did not make it out the Ethernet card. > > > > > > Is this the weirdest thing or what? Ideas would sure be appreciated! > > > > I am assuming there is a typo in your Java version and you are using > Java 8. > > > > Yes, Java 8. > > > That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less > than 6 > > months). If you aren't already planning to upgrade (I'd suggest to > 9.0.x) then > > you might want to start thinking about it. > > > > Vendor constraint. It's a canned application published by a national > software company, and they have not officially approved tomcat 8 for use on > Linux yet. > > > I have a few ideas about what might be going on but rather than fire out > > random theories I have some questions that might help narrow things down. > > > > 1. If this request was successful, how big is the response? > > > > 1035 bytes. > > > 2. If this request was successful, how long would it typically take to > > complete? > > > > Under 60 ms. > > > 3. Looking at the Wireshark trace for a failed request, how long after > the last > > byte of the request is sent by the client does Tomcat send the FIN? > > > > Maybe 100 microseconds. > > > 4. Looking at the Wireshark trace for a failed request, is the request > fully sent > > (including terminating CRLF etc)? > > > > Yes, the request as seen by the tomcat server is complete and is > terminated by 0D 0A. > > > 5. Are there any proxies, firewalls etc between the user agent and > Tomcat? > > > > User agent -> firewall -> nginx plus -> upstream tomcat servers > > > 6. What timeouts are configured for the Connector? > > > > Sorry, which connector are you referring to? > > > 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? > > > > HTTP/1.1 > > > 8. Where are you running Wireshark? User agent? Tomcat? Somewhere > > else? > > On the nginx proxy and both upstream tomcat servers. (On the user agent, > too, but that doesn't help us in this case.) > > If you would like to see a screen shot showing all 4 captures > side-by-size, I can send you a secure link. It will verify my answers > above. It shows 4 separate WireShark captures taken simultaneously: > > (a) the request going from the nginx proxy to tomcat 1 > (b) tomcat 1 receiving the request and terminating the connection > (c) nginx sending the request to tomcat 2 > (d) tomcat 2 replying to the request (but the reply does not help the user > because the tomcat server does not recognize the user agent's JSESSIONID > cookie, so it responds "invalid session." > > > > > Mark > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > Disclaimer : This email and any files transmitted with it are confidential > and intended solely for intended recipients. If you are not the named > addressee you should not disseminate, distribute, copy or alter this email. > Any views or opinions presented in this email are solely those of the > author and might not represent those of Physician Select Management. > Warning: Although Physician Select Management has taken reasonable > precautions to ensure no viruses are present in this email, the company > cannot accept responsibility for any loss or damage arising from the use of > this email or attachments. > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Weirdest Tomcat Behavior Ever?
> -Original Message- > From: Mark Thomas > Sent: Friday, October 16, 2020 5:17 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 16/10/2020 10:05, Eric Robinson wrote: > > Hi Mark -- > > > > Those are great questions. See answers below. > > > > > >> -Original Message- > >> From: Mark Thomas > >> Sent: Friday, October 16, 2020 2:20 AM > >> To: users@tomcat.apache.org > >> Subject: Re: Weirdest Tomcat Behavior Ever? > >> > >> On 16/10/2020 00:27, Eric Robinson wrote: > >> > >> > >> > >>> The localhost_access log shows a request received and an HTTP 200 > >> response sent, as follows... > >>> > >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > >>> /app/code.jsp?gizmoid=64438=5=2020-10- > >> 15 > >>> > >> > lterId=0=0=71340=321072 > >> e > >>> ssid=40696=0.0715816=15102020125245.789063 > HTTP/1.0 > >>> ?gizmoid=64438=5=2020-10- > >> 15=0 > >>> > >> > ionDID=0=71340=321072=40696& > >> rn > >>> d2=0.0715816=15102020125245.789063 200 > >>> > >>> But WireShark shows what really happened. The server received the > >>> GET > >> request, and then it sent a FIN to terminate the connection. So if > >> tomcat sent an HTTP response, it did not make it out the Ethernet card. > >>> > >>> Is this the weirdest thing or what? Ideas would sure be appreciated! > >> > >> I am assuming there is a typo in your Java version and you are using Java > >> 8. > >> > > > > Yes, Java 8. > > > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in > >> less than 6 months). If you aren't already planning to upgrade (I'd > >> suggest to 9.0.x) then you might want to start thinking about it. > >> > > > > Vendor constraint. It's a canned application published by a national > software company, and they have not officially approved tomcat 8 for use on > Linux yet. > > > >> I have a few ideas about what might be going on but rather than fire > >> out random theories I have some questions that might help narrow things > down. > >> > >> 1. If this request was successful, how big is the response? > >> > > > > 1035 bytes. > > > >> 2. If this request was successful, how long would it typically take > >> to complete? > >> > > > > Under 60 ms. > > > >> 3. Looking at the Wireshark trace for a failed request, how long > >> after the last byte of the request is sent by the client does Tomcat send > the FIN? > >> > > > > Maybe 100 microseconds. > > > >> 4. Looking at the Wireshark trace for a failed request, is the > >> request fully sent (including terminating CRLF etc)? > >> > > > > Yes, the request as seen by the tomcat server is complete and is > terminated by 0D 0A. > > > >> 5. Are there any proxies, firewalls etc between the user agent and > Tomcat? > >> > > > > User agent -> firewall -> nginx plus -> upstream tomcat servers > > > >> 6. What timeouts are configured for the Connector? > >> > > > > Sorry, which connector are you referring to? > > > >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? > >> > > > > HTTP/1.1 > > > >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere > >> else? > > > > On the nginx proxy and both upstream tomcat servers. (On the user > > agent, too, but that doesn't help us in this case.) > > > > If you would like to see a screen shot showing all 4 captures side-by-size, > > I > can send you a secure link. It will verify my answers above. It shows 4 > separate WireShark captures taken simultaneously: > > > > (a) the request going from the nginx proxy to tomcat 1 > > (b) tomcat 1 receiving the request and terminating the connection > > (c) nginx sending the request to tomcat 2 > > (d) tomcat 2 replying to the request (but the reply does not help the user > because the tomcat server does not recognize the user agent's JSESSIONID > cookie, so it responds "invalid session." > > Hmm. > > That rules out most of my ideas. > > I'd like to see those screen shots please. Better still would be access to the > captures themselves (just the relevant connections not the whole thing). I > believe what you are telling us but long experience tells me it is best to > double check the original data as well. > I'll send you a link to the screen shot first, then I'll package up the captures and send a link to that in a bit. As the files may contain somewhat sensitive information, I'll send a secure mail direct to your inbox. > I have observed something similar ish in the CI systems. In that case it is > the > requests that disappear. Client side logging shows the request was made but > there is no sign of it ever being received by Tomcat. I don't have network > traces for that (yet) so I'm not sure where the data is going missing. > > I am beginning to suspect there is a hard to trigger Tomcat or JVM bug here. I > think a Tomcat bug is more likely although I have been over the code several > times and I don't see anything. > I'm thinking a bug of some kind, too, but I've been hosting about 1800 instances of tomcat for 15 years and I have
Re: Weirdest Tomcat Behavior Ever?
Hi Eric, These weird situations are sometimes best looked at by confirming what the OS is seeing from user-space. Can you run: sudo strace -r -f -e trace=network -p You can then log that to a file and correlate and see if the kernel is in fact being asked to send the response. It's very insightful to see what is actually going on between the JVM and Kernel. Paul On Fri, Oct 16, 2020 at 12:16 PM Mark Thomas wrote: > On 16/10/2020 10:05, Eric Robinson wrote: > > Hi Mark -- > > > > Those are great questions. See answers below. > > > > > >> -Original Message- > >> From: Mark Thomas > >> Sent: Friday, October 16, 2020 2:20 AM > >> To: users@tomcat.apache.org > >> Subject: Re: Weirdest Tomcat Behavior Ever? > >> > >> On 16/10/2020 00:27, Eric Robinson wrote: > >> > >> > >> > >>> The localhost_access log shows a request received and an HTTP 200 > >> response sent, as follows... > >>> > >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > >>> /app/code.jsp?gizmoid=64438=5=2020-10- > >> 15 > >>> > >> lterId=0=0=71340=321072 > >> e > >>> ssid=40696=0.0715816=15102020125245.789063 HTTP/1.0 > >>> ?gizmoid=64438=5=2020-10- > >> 15=0 > >>> > >> ionDID=0=71340=321072=40696& > >> rn > >>> d2=0.0715816=15102020125245.789063 200 > >>> > >>> But WireShark shows what really happened. The server received the GET > >> request, and then it sent a FIN to terminate the connection. So if > tomcat sent > >> an HTTP response, it did not make it out the Ethernet card. > >>> > >>> Is this the weirdest thing or what? Ideas would sure be appreciated! > >> > >> I am assuming there is a typo in your Java version and you are using > Java 8. > >> > > > > Yes, Java 8. > > > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less > than 6 > >> months). If you aren't already planning to upgrade (I'd suggest to > 9.0.x) then > >> you might want to start thinking about it. > >> > > > > Vendor constraint. It's a canned application published by a national > software company, and they have not officially approved tomcat 8 for use on > Linux yet. > > > >> I have a few ideas about what might be going on but rather than fire out > >> random theories I have some questions that might help narrow things > down. > >> > >> 1. If this request was successful, how big is the response? > >> > > > > 1035 bytes. > > > >> 2. If this request was successful, how long would it typically take to > >> complete? > >> > > > > Under 60 ms. > > > >> 3. Looking at the Wireshark trace for a failed request, how long after > the last > >> byte of the request is sent by the client does Tomcat send the FIN? > >> > > > > Maybe 100 microseconds. > > > >> 4. Looking at the Wireshark trace for a failed request, is the request > fully sent > >> (including terminating CRLF etc)? > >> > > > > Yes, the request as seen by the tomcat server is complete and is > terminated by 0D 0A. > > > >> 5. Are there any proxies, firewalls etc between the user agent and > Tomcat? > >> > > > > User agent -> firewall -> nginx plus -> upstream tomcat servers > > > >> 6. What timeouts are configured for the Connector? > >> > > > > Sorry, which connector are you referring to? > > > >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? > >> > > > > HTTP/1.1 > > > >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere > >> else? > > > > On the nginx proxy and both upstream tomcat servers. (On the user agent, > too, but that doesn't help us in this case.) > > > > If you would like to see a screen shot showing all 4 captures > side-by-size, I can send you a secure link. It will verify my answers > above. It shows 4 separate WireShark captures taken simultaneously: > > > > (a) the request going from the nginx proxy to tomcat 1 > > (b) tomcat 1 receiving the request and terminating the connection > > (c) nginx sending the request to tomcat 2 > > (d) tomcat 2 replying to the request (but the reply does not help the > user because the tomcat server does not recognize the user agent's > JSESSIONID cookie, so it responds "invalid session." > > Hmm. > > That rules out most of my ideas. > > I'd like to see those screen shots please. Better still would be access > to the captures themselves (just the relevant connections not the whole > thing). I believe what you are telling us but long experience tells me > it is best to double check the original data as well. > > I have observed something similar ish in the CI systems. In that case it > is the requests that disappear. Client side logging shows the request > was made but there is no sign of it ever being received by Tomcat. I > don't have network traces for that (yet) so I'm not sure where the data > is going missing. > > I am beginning to suspect there is a hard to trigger Tomcat or JVM bug > here. I think a Tomcat bug is more likely although I have been over the > code several times and I don't see anything. > > A few more questions: > > Which HTTP connector are you using? BIO, NIO or APR/Native? > > Is the
Re: Weirdest Tomcat Behavior Ever?
On 16/10/2020 10:05, Eric Robinson wrote: > Hi Mark -- > > Those are great questions. See answers below. > > >> -Original Message- >> From: Mark Thomas >> Sent: Friday, October 16, 2020 2:20 AM >> To: users@tomcat.apache.org >> Subject: Re: Weirdest Tomcat Behavior Ever? >> >> On 16/10/2020 00:27, Eric Robinson wrote: >> >> >> >>> The localhost_access log shows a request received and an HTTP 200 >> response sent, as follows... >>> >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET >>> /app/code.jsp?gizmoid=64438=5=2020-10- >> 15 >>> >> lterId=0=0=71340=321072 >> e >>> ssid=40696=0.0715816=15102020125245.789063 HTTP/1.0 >>> ?gizmoid=64438=5=2020-10- >> 15=0 >>> >> ionDID=0=71340=321072=40696& >> rn >>> d2=0.0715816=15102020125245.789063 200 >>> >>> But WireShark shows what really happened. The server received the GET >> request, and then it sent a FIN to terminate the connection. So if tomcat >> sent >> an HTTP response, it did not make it out the Ethernet card. >>> >>> Is this the weirdest thing or what? Ideas would sure be appreciated! >> >> I am assuming there is a typo in your Java version and you are using Java 8. >> > > Yes, Java 8. > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less than 6 >> months). If you aren't already planning to upgrade (I'd suggest to 9.0.x) >> then >> you might want to start thinking about it. >> > > Vendor constraint. It's a canned application published by a national software > company, and they have not officially approved tomcat 8 for use on Linux yet. > >> I have a few ideas about what might be going on but rather than fire out >> random theories I have some questions that might help narrow things down. >> >> 1. If this request was successful, how big is the response? >> > > 1035 bytes. > >> 2. If this request was successful, how long would it typically take to >> complete? >> > > Under 60 ms. > >> 3. Looking at the Wireshark trace for a failed request, how long after the >> last >> byte of the request is sent by the client does Tomcat send the FIN? >> > > Maybe 100 microseconds. > >> 4. Looking at the Wireshark trace for a failed request, is the request fully >> sent >> (including terminating CRLF etc)? >> > > Yes, the request as seen by the tomcat server is complete and is terminated > by 0D 0A. > >> 5. Are there any proxies, firewalls etc between the user agent and Tomcat? >> > > User agent -> firewall -> nginx plus -> upstream tomcat servers > >> 6. What timeouts are configured for the Connector? >> > > Sorry, which connector are you referring to? > >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? >> > > HTTP/1.1 > >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere >> else? > > On the nginx proxy and both upstream tomcat servers. (On the user agent, too, > but that doesn't help us in this case.) > > If you would like to see a screen shot showing all 4 captures side-by-size, I > can send you a secure link. It will verify my answers above. It shows 4 > separate WireShark captures taken simultaneously: > > (a) the request going from the nginx proxy to tomcat 1 > (b) tomcat 1 receiving the request and terminating the connection > (c) nginx sending the request to tomcat 2 > (d) tomcat 2 replying to the request (but the reply does not help the user > because the tomcat server does not recognize the user agent's JSESSIONID > cookie, so it responds "invalid session." Hmm. That rules out most of my ideas. I'd like to see those screen shots please. Better still would be access to the captures themselves (just the relevant connections not the whole thing). I believe what you are telling us but long experience tells me it is best to double check the original data as well. I have observed something similar ish in the CI systems. In that case it is the requests that disappear. Client side logging shows the request was made but there is no sign of it ever being received by Tomcat. I don't have network traces for that (yet) so I'm not sure where the data is going missing. I am beginning to suspect there is a hard to trigger Tomcat or JVM bug here. I think a Tomcat bug is more likely although I have been over the code several times and I don't see anything. A few more questions: Which HTTP connector are you using? BIO, NIO or APR/Native? Is the issue reproducible if you switch to a different connector? How easy is it for you to reproduce this issue? How are you linking the request you see in the access log with the request you see in Wireshark? How comfortable are you running a patched version of Tomcat (drop class files provided by me into $CATALINA_BASE/lib in the right directory structure and restart Tomcat)? Just thinking ahead about collecting additional debug information. Thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail:
RE: Weirdest Tomcat Behavior Ever?
> > 6. What timeouts are configured for the Connector? > > > > Sorry, which connector are you referring to? > Sorry, I'm a dummy. Obviously you mean the tomcat connector. connectionTimeout="2" -Eric Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Weirdest Tomcat Behavior Ever?
Hi Mark -- Those are great questions. See answers below. > -Original Message- > From: Mark Thomas > Sent: Friday, October 16, 2020 2:20 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 16/10/2020 00:27, Eric Robinson wrote: > > > > > The localhost_access log shows a request received and an HTTP 200 > response sent, as follows... > > > > 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > > /app/code.jsp?gizmoid=64438=5=2020-10- > 15 > > > lterId=0=0=71340=321072 > e > > ssid=40696=0.0715816=15102020125245.789063 HTTP/1.0 > > ?gizmoid=64438=5=2020-10- > 15=0 > > > ionDID=0=71340=321072=40696& > rn > > d2=0.0715816=15102020125245.789063 200 > > > > But WireShark shows what really happened. The server received the GET > request, and then it sent a FIN to terminate the connection. So if tomcat sent > an HTTP response, it did not make it out the Ethernet card. > > > > Is this the weirdest thing or what? Ideas would sure be appreciated! > > I am assuming there is a typo in your Java version and you are using Java 8. > Yes, Java 8. > That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less than 6 > months). If you aren't already planning to upgrade (I'd suggest to 9.0.x) then > you might want to start thinking about it. > Vendor constraint. It's a canned application published by a national software company, and they have not officially approved tomcat 8 for use on Linux yet. > I have a few ideas about what might be going on but rather than fire out > random theories I have some questions that might help narrow things down. > > 1. If this request was successful, how big is the response? > 1035 bytes. > 2. If this request was successful, how long would it typically take to > complete? > Under 60 ms. > 3. Looking at the Wireshark trace for a failed request, how long after the > last > byte of the request is sent by the client does Tomcat send the FIN? > Maybe 100 microseconds. > 4. Looking at the Wireshark trace for a failed request, is the request fully > sent > (including terminating CRLF etc)? > Yes, the request as seen by the tomcat server is complete and is terminated by 0D 0A. > 5. Are there any proxies, firewalls etc between the user agent and Tomcat? > User agent -> firewall -> nginx plus -> upstream tomcat servers > 6. What timeouts are configured for the Connector? > Sorry, which connector are you referring to? > 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? > HTTP/1.1 > 8. Where are you running Wireshark? User agent? Tomcat? Somewhere > else? On the nginx proxy and both upstream tomcat servers. (On the user agent, too, but that doesn't help us in this case.) If you would like to see a screen shot showing all 4 captures side-by-size, I can send you a secure link. It will verify my answers above. It shows 4 separate WireShark captures taken simultaneously: (a) the request going from the nginx proxy to tomcat 1 (b) tomcat 1 receiving the request and terminating the connection (c) nginx sending the request to tomcat 2 (d) tomcat 2 replying to the request (but the reply does not help the user because the tomcat server does not recognize the user agent's JSESSIONID cookie, so it responds "invalid session." > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weirdest Tomcat Behavior Ever?
On 16/10/2020 00:27, Eric Robinson wrote: > The localhost_access log shows a request received and an HTTP 200 response > sent, as follows... > > 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > /app/code.jsp?gizmoid=64438=5=2020-10-15=0=0=71340=321072=40696=0.0715816=15102020125245.789063 > HTTP/1.0 > ?gizmoid=64438=5=2020-10-15=0=0=71340=321072=40696=0.0715816=15102020125245.789063 > 200 > > But WireShark shows what really happened. The server received the GET > request, and then it sent a FIN to terminate the connection. So if tomcat > sent an HTTP response, it did not make it out the Ethernet card. > > Is this the weirdest thing or what? Ideas would sure be appreciated! I am assuming there is a typo in your Java version and you are using Java 8. That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less than 6 months). If you aren't already planning to upgrade (I'd suggest to 9.0.x) then you might want to start thinking about it. I have a few ideas about what might be going on but rather than fire out random theories I have some questions that might help narrow things down. 1. If this request was successful, how big is the response? 2. If this request was successful, how long would it typically take to complete? 3. Looking at the Wireshark trace for a failed request, how long after the last byte of the request is sent by the client does Tomcat send the FIN? 4. Looking at the Wireshark trace for a failed request, is the request fully sent (including terminating CRLF etc)? 5. Are there any proxies, firewalls etc between the user agent and Tomcat? 6. What timeouts are configured for the Connector? 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? 8. Where are you running Wireshark? User agent? Tomcat? Somewhere else? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
