Re: Question about releases available for download

[Christopher Schultz]

Jon,

On 10/18/23 15:39, Mcalexander, Jon J. wrote:

Thanks Mark. I'm sorry if I stated it incorrectly. I meant the issue
with JDBC being broken, etc. the stuff that prompted the immediate
new releases.

I think the word you were looking for was "regression", not "recursion" ;)

-chris


-Original Message-
From: Mark Thomas 
Sent: Wednesday, October 18, 2023 2:04 PM
To: users@tomcat.apache.org
Subject: Re: Question about releases available for download

On 18/10/2023 18:29, Mcalexander, Jon J. wrote:

Hi Mark, et-al,

With the recursion error with these releases in mind, should 8.5.94, 9.0.81,

and 10.1.15 be available for download via the archives? Should they not be
removed and a not placed in the location that they have been removed due
to introduced issues?

Recursion error?

Regardless, all ASF releases will always be available from the ASF archives.
That is ASF policv and I don't see it changing.

Yes, old releases have bugs and/or security issues. Yes, some of those bugs /
security issues are quite nasty. That is why we always recommend using the
latest release of a current supported major version.

Maven Central has a similar policy. Once a release is published to Maven
Central it is pretty much impossible to get it removed.

Mark



Just asking,

Thanks.

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508



jonmcalexan...@wellsfargo.com

This message may contain confidential and/or privileged information. If you

are not the addressee or authorized to receive this for the addressee, you
must not use, copy, disclose, or take any action based on this message or any
information herein. If you have received this message in error, please advise
the sender immediately by reply e-mail and delete this message. Thank you
for your cooperation.





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9 -> Intermittent 404 (3-4 fails in 20-30 million requests daily sometimes )

[Christopher Schultz]

Anurag,

On 10/17/23 10:01, Anurag Kumar wrote:

Thanks, Christopher, for looking into this issue.


Wait until I actually help before thanking me. I'm mostly trying to get 
more information so people smarter than I am can maybe help you. ;)



Tomcat version:
Server version: Apache Tomcat/9.0.74
Server built: Apr 13, 2023 08:10:39 UTC
Server number: 9.0.74.0


Would it be at all possible to upgrade to 9.0.latest on one or more of 
your cluster members? You'd have to read the changelog to see if there 
are any incompatibilities but I suspect you should be okay. There are 
constant improvements, and its possible that 6 months (since 9.0.74) has 
improved something that makes a difference for you.


We became aware of this issue a few days ago when it was reported by a 
customer due to a critical internal API failure, where the possibility 
of unexpected characters was none. Upon investigating the Splunk logs, 
we discovered that this issue had been occurring for at least the past 3 
months, based on the available three-month log data.


We have a single servlet mapped for all URL patterns, and we log the 
requests from this servlet. Internally, we always return a 200 response 
code with the appropriate error page and never throw a 404 response.


Do you ever re-deploy your application while Tomcat is running? Which 
file/log contains the 404 responses?



Here is our Connector configuration:

scheme="https" secure="true" executor="httpsThreadPool" 
acceptCount="250" SSLEnabled="false" connectionTimeout="2" 
URIEncoding="utf-8" enableLookups="false" 
relaxedQueryChars="{}|<>"" compression="on"/>



These 404 issues have been observed on requests created from Chrome, 
HttpURLConnection in Java, and AsyncHttpClient in Java. Our servers are 
behind an Amazon Load Balancer (ALB), and while ALB operates on HTTP2, 
our Tomcat servers are configured for HTTP1.


This issue has been reported on all nine different clusters running the 
same Tomcat version. Our test environment closely mirrors the production 
environment, but we have been unable to reproduce the issue so far, even 
after increasing the number of requests.


It's challenging to identify any specific patterns as the occurrences 
appear to be distributed randomly and happen with very simple GET 
requests. There was one instance where I was able to reproduce the issue 
in production with a straightforward GET request after making 45,000 
calls, but it was never reproduced afterwards through my automation.


:/


Request capture on ALB:-
image.png


This list strips images out. Please send plain-text only.

-chris

On Mon, Oct 16, 2023 at 6:16 PM Christopher Schultz 
mailto:ch...@christopherschultz.net>> wrote:


Anurag,

On 10/15/23 04:48, Anurag Kumar wrote:
 >
 > Hi, we are experiencing intermittent 404 errors with both GET and
POST
 > calls. These errors are quite rare and have proven difficult to
 > reproduce in our testing environment. However, on our production
system,
 > we encounter 3-4 cases daily out of 20-30 million requests where
a 404
 > error appears in the Tomcat access logs, and the corresponding call
 > fails to reach the mapped servlet. Interestingly, the same calls
work
 > perfectly just a few milliseconds before and after on the same node.
 > This inconsistency is causing significant issues, especially when
 > critical API calls fail and are not automatically retried.
 >
 > Is there any open issue related to this problem that we should be
aware of?

None that I know of personally.

Can you post your exact Tomcat version, your  configuration
with any secrets removed and a little more background on the type of
traffic you are seeing (e.g. HTTP/1.1 v h2, TLS or not, etc.). Are you
able to tell if these failed requests are part of any kind of pipelined
requests (HTTP Keep-Alive) or h2 single channels?

Understanding the network topology may be relevant, though its unlikely
that any lb/rp is doing this, as you can see the logs on the Tomcat
node. But it may change the way the requests are being handled based
upon the type of connection between the lb/rp and Tomcat.

Have you double-checked that the URIs are clean and don't contain
anything unexpected such as lookalike characters, etc.? I suspect this
is not an issue since you said "critical API calls fail" which leads me
to understand that you have legitimate customers reporting these
failures, instead of just investigating unexpected entries in your log
files.

Is your testing environment reasonably similar to production? What
would
happen if you were to reply a whole day's worth of production-requests
through your testing environment?

Is there any pattern whatsoever in the failed requests? If you look at
every failed request for all time, are they randomly distributed
throughout your URI space, or do you find that som

Re: Tomcat minor update

[Christopher Schultz]

Mark and Aditya,

On 10/18/23 04:21, Mark Thomas wrote:

On 17/10/2023 22:47, Aditya Shastri wrote:

Hello,

We have several tomcat instances that use a single CATALINA_HOME which
is a symlink for a specific version. The Tomcat instance we use is
very barebones and doesn't have any of the apps that come with it.

For example,
The CATALINA_HOME points to a symlink
/opt/tomcat/tomcat-9/tomcat-9-latest ->
/opt/tomcat/tomcat-9/apache-tomcat-9.0.80.

Now, if I want to upgrade to apache-tomcat-9.0.82, I normally do the
following steps:
1. Stop all the running instances of tomcat in the various 
'CATALINA_BASE'.

2. Update the symlink /opt/tomcat/tomcat-9/tomcat-9-latest from
/opt/tomcat/tomcat-9/apache-tomcat-9.0.80 to
/opt/tomcat/tomcat-9/apache-tomcat-9.0.82.
3. Start all the instances.

This method appears to work, and I read it as the most appropriate 
method.


My question is, can I change the symlink,
'/opt/tomcat/tomcat-9/tomcat-9-latest', while all the instances are
running and restart the instances when I have downtime?


Probably not. You might get away with it sometimes but sometimes you are 
going to see errors.



Does Tomcat load all the CATALINA_HOME jar(s) (not including the
webapps folder) and config to memory thereby not caring if the
libraries have changed or does it realize that something has changed?


No. The JVM loads classes when they are first referenced.

The issue will be if you update the symlink, then Tomcat tries to load 
another class and the class from the new version is not compatible with 
the classes from the old version. A failure is unlikely but not 
impossible. I wouldn't risk it.


I wonder if we could solve this, at least on *NIX, by resolving 
CATALINA_HOME by using `readlink -f`. This would allow you to use a 
symlink to point to Tomcat but after catalina.sh is invoked, 
CATALINA_HOME could be replaced with a canonicalized one which does not 
contain a symlink anymore. Maybe there is a similar 
utility/command/path-mangling-magic available on Windows?


That would allow you to change the symlink and not disturb any 
currently-running Tomcat instances. You would obviously not want to 
remove the old version from the disk before shutting-down those 
instances, of course.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Stale tomcat.pid file prevented Tomcat from starting

[Christopher Schultz]

Darryl,

On 10/17/23 10:30, Darryl Baker wrote:

We are running 9.0.78 on RHEL 7. During our monthly patch and reboot cycle one 
the Tomcat running on one system failed to restart. The error said that there 
was a running version of Tomcat with a low PID number. Just rerunning the start 
“systemctl start tomcat” solved the issue. We use the default PID file. I 
looked at the Catalina.sh script and there is a bunch of logic trying to 
prevent this issue but somehow that failed. Is there any know issues with this 
logic?


I'm sure it's gone, now, but did you check the timestamp(s) on the PID 
file? They may have given you a clue as to whether it was leftover from 
a previous boot cycle, or if Tomcat actually failed to startup during 
*this* reboot cycle.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Question about releases available for download

[Mcalexander, Jon J.]

Thanks Mark. I'm sorry if I stated it incorrectly. I meant the issue with JDBC 
being broken, etc. the stuff that prompted the immediate new releases.

Thank you!

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.

> -Original Message-
> From: Mark Thomas 
> Sent: Wednesday, October 18, 2023 2:04 PM
> To: users@tomcat.apache.org
> Subject: Re: Question about releases available for download
> 
> On 18/10/2023 18:29, Mcalexander, Jon J. wrote:
> > Hi Mark, et-al,
> >
> > With the recursion error with these releases in mind, should 8.5.94, 9.0.81,
> and 10.1.15 be available for download via the archives? Should they not be
> removed and a not placed in the location that they have been removed due
> to introduced issues?
> 
> Recursion error?
> 
> Regardless, all ASF releases will always be available from the ASF archives.
> That is ASF policv and I don't see it changing.
> 
> Yes, old releases have bugs and/or security issues. Yes, some of those bugs /
> security issues are quite nasty. That is why we always recommend using the
> latest release of a current supported major version.
> 
> Maven Central has a similar policy. Once a release is published to Maven
> Central it is pretty much impossible to get it removed.
> 
> Mark
> 
> >
> > Just asking,
> >
> > Thanks.
> >
> > Dream * Excel * Explore * Inspire
> > Jon McAlexander
> > Senior Infrastructure Engineer
> > Asst. Vice President
> > He/His
> >
> > Middleware Product Engineering
> > Enterprise CIO | EAS | Middleware | Infrastructure Solutions
> >
> > 8080 Cobblestone Rd | Urbandale, IA 50322
> > MAC: F4469-010
> > Tel 515-988-2508 | Cell 515-988-2508
> >
> >
> jonmcalexan...@wellsfargo.com
> > This message may contain confidential and/or privileged information. If you
> are not the addressee or authorized to receive this for the addressee, you
> must not use, copy, disclose, or take any action based on this message or any
> information herein. If you have received this message in error, please advise
> the sender immediately by reply e-mail and delete this message. Thank you
> for your cooperation.
> >
> >
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Question about releases available for download

[Mark Thomas]

On 18/10/2023 18:29, Mcalexander, Jon J. wrote:

Hi Mark, et-al,

With the recursion error with these releases in mind, should 8.5.94, 9.0.81, 
and 10.1.15 be available for download via the archives? Should they not be 
removed and a not placed in the location that they have been removed due to 
introduced issues?


Recursion error?

Regardless, all ASF releases will always be available from the ASF 
archives. That is ASF policv and I don't see it changing.


Yes, old releases have bugs and/or security issues. Yes, some of those 
bugs / security issues are quite nasty. That is why we always recommend 
using the latest release of a current supported major version.


Maven Central has a similar policy. Once a release is published to Maven 
Central it is pretty much impossible to get it removed.


Mark



Just asking,

Thanks.

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Question about releases available for download

[Mcalexander, Jon J.]

Hi Mark, et-al,

With the recursion error with these releases in mind, should 8.5.94, 9.0.81, 
and 10.1.15 be available for download via the archives? Should they not be 
removed and a not placed in the location that they have been removed due to 
introduced issues?

Just asking,

Thanks.

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.

Is RMI leak prevention working properly? Shutdown failure

[Leon Atherton]

Hello,

I have observed that the Tomcat process does not shut down properly when 
the RMI leak prevention is triggered. The process remains alive and 
holds onto the RMI port.


I have created this sample project to demonstrate the issue:
https://github.com/leonatherton/rmi-leak-test/blob/master/src/main/java/rmi/RmiLeakTestServlet.java

There are prebuilt wars here:
https://github.com/leonatherton/rmi-leak-test/releases/tag/v0.0.0

Steps to reproduce:
1. Deploy the war (rmi-leak-test.war)
2. Trigger a shutdown
3. Observe that the process does not end, and holds on to the RMI port 
(can be seen with netstat -tulpn)


This message is logged:

SEVERE [main] 
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets 
Found RMI Target with stub class class [com.sun.proxy.$Proxy5] and value 
[Proxy[MyRemote,RemoteObjectInvocationHandler[UnicastRef [liveRef: 
[endpoint:[127.0.0.1:46013](local),objID:[-30127669:18b42a65f21:-7fff, 
8331338290308659576]]. This RMI Target has been forcibly removed to 
prevent a memory leak.


Tomcat obviously tries to tidy up, but it seems like maybe not 
successfully. I am wondering if it is expected behaviour for the process 
to remain alive?


Full log below.

Thanks,
Leon

NOTE: Picked up JDK_JAVA_OPTIONS: 
--add-opens=java.base/java.lang=ALL-UNNAMED 
--add-opens=java.base/java.io=ALL-UNNAMED 
--add-opens=java.base/java.util=ALL-UNNAMED 
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED 
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
18-Oct-2023 11:57:59.621 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version 
name:   Apache Tomcat/9.0.73
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server 
built:  Feb 27 2023 15:33:40 UTC
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version 
number: 9.0.73.0
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS 
Name:   Linux
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS 
Version:    3.10.0-1160.45.1.el7.x86_64
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log 
Architecture:  amd64
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Java 
Home: /usr/lib/jvm/java-11-amazon-corretto
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM 
Version:   11.0.20.1+9-LTS
18-Oct-2023 11:57:59.650 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM 
Vendor:    Amazon.com Inc.
18-Oct-2023 11:57:59.651 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log 
CATALINA_BASE: /home/converter/apache-tomcat-base
18-Oct-2023 11:57:59.651 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log 
CATALINA_HOME: /home/converter/apache-tomcat-9.0.73
18-Oct-2023 11:57:59.669 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: --add-opens=java.base/java.lang=ALL-UNNAMED
18-Oct-2023 11:57:59.669 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: --add-opens=java.base/java.io=ALL-UNNAMED
18-Oct-2023 11:57:59.669 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: --add-opens=java.base/java.util=ALL-UNNAMED
18-Oct-2023 11:57:59.669 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
18-Oct-2023 11:57:59.669 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
18-Oct-2023 11:57:59.669 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: 
-Djava.util.logging.config.file=/home/converter/apache-tomcat-base//conf/logging.properties
18-Oct-2023 11:57:59.670 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
18-Oct-2023 11:57:59.670 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: -Djdk.tls.ephemeralDHKeySize=2048
18-Oct-2023 11:57:59.670 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
18-Oct-2023 11:57:59.670 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
18-Oct-2023 11:57:59.670 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: -Dignore.endorsed.dirs=
18-Oct-2023 11:57:59.670 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line 
argument: -Dca

Re: Tomcat minor update

[Mark Thomas]

On 17/10/2023 22:47, Aditya Shastri wrote:

Hello,

We have several tomcat instances that use a single CATALINA_HOME which
is a symlink for a specific version. The Tomcat instance we use is
very barebones and doesn't have any of the apps that come with it.

For example,
The CATALINA_HOME points to a symlink
/opt/tomcat/tomcat-9/tomcat-9-latest ->
/opt/tomcat/tomcat-9/apache-tomcat-9.0.80.

Now, if I want to upgrade to apache-tomcat-9.0.82, I normally do the
following steps:
1. Stop all the running instances of tomcat in the various 'CATALINA_BASE'.
2. Update the symlink /opt/tomcat/tomcat-9/tomcat-9-latest from
/opt/tomcat/tomcat-9/apache-tomcat-9.0.80 to
/opt/tomcat/tomcat-9/apache-tomcat-9.0.82.
3. Start all the instances.

This method appears to work, and I read it as the most appropriate method.

My question is, can I change the symlink,
'/opt/tomcat/tomcat-9/tomcat-9-latest', while all the instances are
running and restart the instances when I have downtime?


Probably not. You might get away with it sometimes but sometimes you are 
going to see errors.



Does Tomcat load all the CATALINA_HOME jar(s) (not including the
webapps folder) and config to memory thereby not caring if the
libraries have changed or does it realize that something has changed?


No. The JVM loads classes when they are first referenced.

The issue will be if you update the symlink, then Tomcat tries to load 
another class and the class from the new version is not compatible with 
the classes from the old version. A failure is unlikely but not 
impossible. I wouldn't risk it.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org