Re: Tomcat/Java starts using too much memory and not by the heap or non-heap memory

[Christopher Schultz]

Brian,

On 1/5/24 17:21, Brian Braun wrote:

Hello Chirstopher,

First of all: thanks a lot for your responses!

On Wed, Jan 3, 2024 at 9:25 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:


Brian,

On 12/30/23 15:42, Brian Braun wrote:

At the beginning, this was the problem: The OOM-killer (something that I
never knew existed) killing Tomcat unexpectedly and without any
explanation


The explanation is always the same: some application requests memory
from the kernel, which always grants the request(!). When the
application tries to use that memory, the kernel scrambles to physically
allocate the memory on-demand and, if all the memory is gone, it will
pick a process and kill it.

>

Yes, that was happening to me until I set up the SWAP file and now at least
the Tomcat process is not being killed anymore.


Swap can get you out of a bind like this, but it will ruin your 
performance. If you care more about stability (and believe me, it's a 
reasonable decision), then leave the swap on. But swap will kill (a) 
performance (b) SSD lifetime and (c) storage/transaction costs depending 
upon your environment. Besides, you either need the memory or you do 
not. It's rare to "sometimes" need the memory.



Using a swap file is probably going to kill your performance. What
happens if you make your heap smaller?

>

Yes, in fact the performance is suffering and that is why I don't consider
the swap file as a solution.


:D


I have assigned to -Xmx both small amounts (as small as 300MB) and high
amounts (as high as 1GB) and the problem is still present (the Tomcat
process grows in memory usage up to 1.5GB combining real memory and swap
memory).


Okay, that definitely indicates a problem that needs to be solved.

I've seen things like native ZIP handling code leaking native memory, 
but I know that Tomcat does not leak like that. If you do anything in 
your application that might leave file handles open, it could be 
contributing to the problem.



As I have explained in another email recently, I think that neither heap
usage nor non-heap usage are the problem. I have been monitoring them and
their requirements have always stayed low enough, so I could leave the -Xms
parameter with about 300-400 MB and that would be enough.


Well, between heap and non-heap, that's all the memory. There is no 
non-heap-non-non-heap memory to be counted. Technically stack space is 
the same as "native memory" but usually you experience other problems if 
you have too many threads and they are running out of stack space.



There is something else in the JVM that is using all that memory and I
still don't know what it is. And I think it doesn't care about the value I
give to -Xmx, it uses all the memory it wants. Doing what? I don't know.


It might be time to start digging into those native memory-tracking tools.


Maybe I am not understanding your suggestion.
I have assigned to -Xmx both small amounts (as small as 300MB) and high
amounts (as high as 1GB) and the problem is still present. In fact the
problem started with a low amount for -Xmx.


No, you are understanding my suggestion(s). But if you are hitting Linux 
oom-killer with a 300MiB heap and a process size that is growing to 1.5G 
then getting killed... it's time to dig deeper.


-chris


On Sat, Dec 30, 2023 at 12:44 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:


Brian,

On 12/29/23 20:48, Brian Braun wrote:

Hello,

First of all:
Christopher Schultz: You answered an email from me 6 weeks ago. You

helped

me a lot with your suggestions. I have done a lot of research and have
learnt a lot since then, so I have been able to rule out a lot of

potential

roots for my issue. Because of that I am able to post a new more

specific

email. Thanks a lot!!!

Now, this is my stack:

- Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been

enough

for years.
- Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1

2023-08-24

- Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx1000m

-Xms1000m

..")
- My app, which I developed myself, and has been running without any
problems for years

Well, a couple of months ago my website/Tomcat/Java started eating more

and

more memory about after about 4-7 days. The previous days it uses just

a

few hundred MB and is very steady, but then after a few days the memory
usage suddenly grows up to 1.5GB (and then stops growing at that point,
which is interesting). Between these anomalies the RAM usage is fine

and

very steady (as it has been for years) and it uses just about 40-50% of

the

"Max memory" (according to what the Tomcat Manager server status

shows).

The 3 components of G1GC heap memory are steady and low, before and

after

the usage grows to 1.5GB, so it is definitely not that the heap starts
requiring more and more memory. I have been using several tools to

monitor

that (New Relic, VisualVM and JDK Mission Control) so I'm sure that the
memory usage by the heap is 

Re: Secure connection - only ipv6 listener at port 8443

[Christoph Kukulies]

Ubuntu 22.04.3
Thanks. Output is indeed "0". So it listens to both protocol versions. Will 
stick with this of course :)

--
Christoph


> Am 08.01.2024 um 11:15 schrieb EML :
> 
> Run this command (I'm assuming you're on Linux):
> 
>  $ cat /proc/sys/net/ipv6/bindv6only
> 
> The default output is 0 on Linux (the Windows equivalent is apparently 1). 0 
> means that an IPv6 socket will connect to *both* IPv6 and IPv4  (look up 
> IPV6_V6ONLY).
> 
> You can force Java to listen to IPv4 only, with something like 
> 'Environment="JAVA_OPTS_APPEND=-Djava.net.preferIPv4Stack=true', but you 
> probably don't want to do this.
> 
> On 08/01/2024 09:40, Christoph Kukulies wrote:
>> Hi,
>> 
>> I'm trying to connect to a tomcat9 server for which I made an entry in 
>> server.xml:
>> 
>>  >maxThreads="150" SSLEnabled="true" >
>>   
>> 
>>   >  certificateFile="conf/localhost-rsa-cert.pem"
>>  certificateChainFile="conf/localhost-rsa-chain.pem"
>>  type="RSA" />
>> 
>> 
>> 
>> and the following entries are also present, FWIW:
>> 
>> >  proxyName="cms.mysite.de  > >" maxThreads="150" enableLookups="false"
>>  redirectPort="8443" acceptCount="100" connectionTimeout="2"
>>  disableUploadTimeout="true" />
>> >  proxyName="othername.de  > >" maxThreads="150" enableLookups="false"
>>  redirectPort="8443" acceptCount="100" connectionTimeout="2"
>>  disableUploadTimeout="true" />
>> >  proxyName="other.de  > >" maxThreads="150" enableLookups="false"
>>  redirectPort="8443" acceptCount="100" connectionTimeout="2"
>>  disableUploadTimeout="true" />
>> >  proxyName="alternate.de  > >" maxThreads="150" enableLookups="false"
>>  redirectPort="8443" acceptCount="100" connectionTimeout="2"
>>  disableUploadTimeout="true" />
>> 
>> 
>> Connection to the server through port 8443 seems to work (filtered by ufw)
>> but I'm wondering why I don't see any tcp/v4 connection.
>> 
>> root@mail:/var/lib/tomcat9/logs# ufw status numbered
>> Status: active
>> 
>>   To Action  From
>>   -- --  
>> [ 1] 25 ALLOW INAnywhere
>> [ 2] 443ALLOW INAnywhere
>> [ 3] 993ALLOW INAnywhere
>> [ 4] 587ALLOW INAnywhere
>> [ 5] 80 ALLOW INAnywhere
>> [ 6] 22 ALLOW IN33.180.24.29
>> [ 7] 8080/tcp   ALLOW IN33.180.24.29
>> [ 8] 8443/tcp   ALLOW IN33.180.24.29
>> [ 9] 25 (v6)ALLOW INAnywhere (v6)
>> [10] 443 (v6)   ALLOW INAnywhere (v6)
>> [11] 993 (v6)   ALLOW INAnywhere (v6)
>> [12] 587 (v6)   ALLOW INAnywhere (v6)
>> [13] 80 (v6)ALLOW INAnywhere (v6)
>> 
>> root@mail:/var/lib/tomcat9/logs# netstat -an | grep 8443
>> tcp6   0  0 :::*8443*:::*LISTEN
>> 08-Jan-2024 10:25:55.513 INFO [main] 
>> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet 
>> engine: [Apache Tomcat/9.0.58 (Ubuntu)]
>> 08-Jan-2024 10:25:55.532 INFO [main] 
>> org.apache.catalina.startup.HostConfig.deployWAR Deploying web application 
>> archive [/var/lib/tomcat9/webapps/ROOT.war]
>> 08-Jan-2024 10:26:07.487 INFO [main] 
>> org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned 
>> for TLDs yet contained no TLDs. Enable debug logging for this logger for a 
>> complete list of JARs that were scanned but no TLDs were found in them. 
>> Skipping unneeded JARs during scanning can improve startup time and JSP 
>> compilation time.
>> 08-Jan-2024 10:26:23.038 INFO [main] 
>> org.apache.catalina.startup.HostConfig.deployWAR Deployment of web 
>> application archive [/var/lib/tomcat9/webapps/ROOT.war] has finished in 
>> [27,506] ms
>> 08-Jan-2024 10:26:23.043 INFO [main] 
>> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
>> ["http-nio-8082"]
>> 08-Jan-2024 10:26:23.079 INFO [main] 
>> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
>> ["http-nio-8081"]
>> 08-Jan-2024 10:26:23.094 INFO [main] 
>> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
>> ["http-nio-8080"]
>> 08-Jan-2024 10:26:23.098 INFO [main] 
>> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
>> ["http-nio-8086"]
>> 08-Jan-2024 10:26:23.108 INFO [main] 
>> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
>> ["https-openssl-apr-8443"]
>> 08-Jan-2024 10:26:23.133 INFO [main] 
>> org.apache.catalina.startup.Catalina.start Server 

Re: Secure connection - only ipv6 listener at port 8443

[EML]

Run this command (I'm assuming you're on Linux):

 $ cat /proc/sys/net/ipv6/bindv6only

The default output is 0 on Linux (the Windows equivalent is apparently 
1). 0 means that an IPv6 socket will connect to *both* IPv6 and IPv4  
(look up IPV6_V6ONLY).


You can force Java to listen to IPv4 only, with something like 
'Environment="JAVA_OPTS_APPEND=-Djava.net.preferIPv4Stack=true', but you 
probably don't want to do this.


On 08/01/2024 09:40, Christoph Kukulies wrote:

Hi,

I'm trying to connect to a tomcat9 server for which I made an entry in 
server.xml:


 
  
        
  
        
            

and the following entries are also present, FWIW:

     proxyName="cms.mysite.de 
" maxThreads="150" enableLookups="false"

 redirectPort="8443" acceptCount="100" connectionTimeout="2"
     disableUploadTimeout="true" />
     proxyName="othername.de 
" maxThreads="150" enableLookups="false"

 redirectPort="8443" acceptCount="100" connectionTimeout="2"
     disableUploadTimeout="true" />
     proxyName="other.de 
" maxThreads="150" enableLookups="false"

 redirectPort="8443" acceptCount="100" connectionTimeout="2"
     disableUploadTimeout="true" />
     proxyName="alternate.de 
" maxThreads="150" enableLookups="false"

 redirectPort="8443" acceptCount="100" connectionTimeout="2"
     disableUploadTimeout="true" />


Connection to the server through port 8443 seems to work (filtered by ufw)
but I'm wondering why I don't see any tcp/v4 connection.

root@mail:/var/lib/tomcat9/logs# ufw status numbered
Status: active

  To                         Action      From
  --                         --      
[ 1] 25                         ALLOW IN    Anywhere
[ 2] 443                        ALLOW IN    Anywhere
[ 3] 993                        ALLOW IN    Anywhere
[ 4] 587                        ALLOW IN    Anywhere
[ 5] 80                         ALLOW IN    Anywhere
[ 6] 22                         ALLOW IN    33.180.24.29
[ 7] 8080/tcp                   ALLOW IN    33.180.24.29
[ 8] 8443/tcp                   ALLOW IN    33.180.24.29
[ 9] 25 (v6)                    ALLOW IN    Anywhere (v6)
[10] 443 (v6)                   ALLOW IN    Anywhere (v6)
[11] 993 (v6)                   ALLOW IN    Anywhere (v6)
[12] 587 (v6)                   ALLOW IN    Anywhere (v6)
[13] 80 (v6)                    ALLOW IN    Anywhere (v6)

root@mail:/var/lib/tomcat9/logs# netstat -an | grep 8443
tcp6       0      0 :::*8443*                :::*                    
LISTEN
08-Jan-2024 10:25:55.513 INFO [main] 
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet 
engine: [Apache Tomcat/9.0.58 (Ubuntu)]
08-Jan-2024 10:25:55.532 INFO [main] 
org.apache.catalina.startup.HostConfig.deployWAR Deploying web 
application archive [/var/lib/tomcat9/webapps/ROOT.war]
08-Jan-2024 10:26:07.487 INFO [main] 
org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was 
scanned for TLDs yet contained no TLDs. Enable debug logging for this 
logger for a complete list of JARs that were scanned but no TLDs were 
found in them. Skipping unneeded JARs during scanning can improve 
startup time and JSP compilation time.
08-Jan-2024 10:26:23.038 INFO [main] 
org.apache.catalina.startup.HostConfig.deployWAR Deployment of web 
application archive [/var/lib/tomcat9/webapps/ROOT.war] has finished 
in [27,506] ms
08-Jan-2024 10:26:23.043 INFO [main] 
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
["http-nio-8082"]
08-Jan-2024 10:26:23.079 INFO [main] 
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
["http-nio-8081"]
08-Jan-2024 10:26:23.094 INFO [main] 
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
["http-nio-8080"]
08-Jan-2024 10:26:23.098 INFO [main] 
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
["http-nio-8086"]
08-Jan-2024 10:26:23.108 INFO [main] 
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
["https-openssl-apr-8443"]
08-Jan-2024 10:26:23.133 INFO [main] 
org.apache.catalina.startup.Catalina.start Server startup in [27735] 
milliseconds


root@mail:/var/lib/tomcat9/logs# lsof -i :8443
COMMAND   PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java   177550 tomcat   43u  IPv6 1853946      0t0  TCP *:8443 (LISTEN)
root@mail:/var/lib/tomcat9/logs#

--
Christoph

Secure connection - only ipv6 listener at port 8443

[Christoph Kukulies]

Hi,

I'm trying to connect to a tomcat9 server for which I made an entry in 
server.xml:

   






and the following entries are also present, FWIW:







Connection to the server through port 8443 seems to work (filtered by ufw)
but I'm wondering why I don't see any tcp/v4 connection.

root@mail:/var/lib/tomcat9/logs# ufw status numbered
Status: active

 To Action  From
 -- --  
[ 1] 25 ALLOW INAnywhere  
[ 2] 443ALLOW INAnywhere  
[ 3] 993ALLOW INAnywhere  
[ 4] 587ALLOW INAnywhere  
[ 5] 80 ALLOW INAnywhere  
[ 6] 22 ALLOW IN33.180.24.29 
[ 7] 8080/tcp   ALLOW IN33.180.24.29 
[ 8] 8443/tcp   ALLOW IN33.180.24.29 
[ 9] 25 (v6)ALLOW INAnywhere (v6) 
[10] 443 (v6)   ALLOW INAnywhere (v6) 
[11] 993 (v6)   ALLOW INAnywhere (v6) 
[12] 587 (v6)   ALLOW INAnywhere (v6) 
[13] 80 (v6)ALLOW INAnywhere (v6) 

root@mail:/var/lib/tomcat9/logs# netstat -an | grep 8443
tcp6   0  0 :::8443 :::*LISTEN 
08-Jan-2024 10:25:55.513 INFO [main] 
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: 
[Apache Tomcat/9.0.58 (Ubuntu)]
08-Jan-2024 10:25:55.532 INFO [main] 
org.apache.catalina.startup.HostConfig.deployWAR Deploying web application 
archive [/var/lib/tomcat9/webapps/ROOT.war]
08-Jan-2024 10:26:07.487 INFO [main] 
org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for 
TLDs yet contained no TLDs. Enable debug logging for this logger for a complete 
list of JARs that were scanned but no TLDs were found in them. Skipping 
unneeded JARs during scanning can improve startup time and JSP compilation time.
08-Jan-2024 10:26:23.038 INFO [main] 
org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application 
archive [/var/lib/tomcat9/webapps/ROOT.war] has finished in [27,506] ms
08-Jan-2024 10:26:23.043 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler ["http-nio-8082"]
08-Jan-2024 10:26:23.079 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler ["http-nio-8081"]
08-Jan-2024 10:26:23.094 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler ["http-nio-8080"]
08-Jan-2024 10:26:23.098 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler ["http-nio-8086"]
08-Jan-2024 10:26:23.108 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler ["https-openssl-apr-8443"]
08-Jan-2024 10:26:23.133 INFO [main] org.apache.catalina.startup.Catalina.start 
Server startup in [27735] milliseconds

root@mail:/var/lib/tomcat9/logs# lsof -i :8443
COMMANDPID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java177550 tomcat   43u  IPv6 1853946  0t0  TCP *:8443 (LISTEN)
root@mail:/var/lib/tomcat9/logs# 

--
Christoph



smime.p7s
Description: S/MIME cryptographic signature

Re: EOL - Tomcat versions

[Mark Thomas]

On 08/01/2024 06:47, i...@flyingfischer.ch wrote:

https://endoflife.date/tomcat

Am 08.01.24 um 07:39 schrieb Deshmukh, Kedar:

Hello,

Could you please throw some light on Tomcat versions and its EOL plan?


See https://tomcat.apache.org/whichversion.html


   1.  8.5.X


EOL 31 March 2024


   2.  9.0.X


No plans.
See https://lists.apache.org/thread/qlzpscgoqct9wspkj5qjkm34s66jswj0


   3.  10.0.X


Already EOL as of 31 October 2022


   4.  10.1.X


No plans.
See https://lists.apache.org/thread/qlzpscgoqct9wspkj5qjkm34s66jswj0

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org