Brian,
On 1/5/24 17:21, Brian Braun wrote:
Hello Chirstopher,
First of all: thanks a lot for your responses!
On Wed, Jan 3, 2024 at 9:25 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:
Brian,
On 12/30/23 15:42, Brian Braun wrote:
At the beginning, this was the problem: The OOM-killer (something that I
never knew existed) killing Tomcat unexpectedly and without any
explanation
The explanation is always the same: some application requests memory
from the kernel, which always grants the request(!). When the
application tries to use that memory, the kernel scrambles to physically
allocate the memory on-demand and, if all the memory is gone, it will
pick a process and kill it.
>
Yes, that was happening to me until I set up the SWAP file and now at least
the Tomcat process is not being killed anymore.
Swap can get you out of a bind like this, but it will ruin your
performance. If you care more about stability (and believe me, it's a
reasonable decision), then leave the swap on. But swap will kill (a)
performance (b) SSD lifetime and (c) storage/transaction costs depending
upon your environment. Besides, you either need the memory or you do
not. It's rare to "sometimes" need the memory.
Using a swap file is probably going to kill your performance. What
happens if you make your heap smaller?
>
Yes, in fact the performance is suffering and that is why I don't consider
the swap file as a solution.
:D
I have assigned to -Xmx both small amounts (as small as 300MB) and high
amounts (as high as 1GB) and the problem is still present (the Tomcat
process grows in memory usage up to 1.5GB combining real memory and swap
memory).
Okay, that definitely indicates a problem that needs to be solved.
I've seen things like native ZIP handling code leaking native memory,
but I know that Tomcat does not leak like that. If you do anything in
your application that might leave file handles open, it could be
contributing to the problem.
As I have explained in another email recently, I think that neither heap
usage nor non-heap usage are the problem. I have been monitoring them and
their requirements have always stayed low enough, so I could leave the -Xms
parameter with about 300-400 MB and that would be enough.
Well, between heap and non-heap, that's all the memory. There is no
non-heap-non-non-heap memory to be counted. Technically stack space is
the same as "native memory" but usually you experience other problems if
you have too many threads and they are running out of stack space.
There is something else in the JVM that is using all that memory and I
still don't know what it is. And I think it doesn't care about the value I
give to -Xmx, it uses all the memory it wants. Doing what? I don't know.
It might be time to start digging into those native memory-tracking tools.
Maybe I am not understanding your suggestion.
I have assigned to -Xmx both small amounts (as small as 300MB) and high
amounts (as high as 1GB) and the problem is still present. In fact the
problem started with a low amount for -Xmx.
No, you are understanding my suggestion(s). But if you are hitting Linux
oom-killer with a 300MiB heap and a process size that is growing to 1.5G
then getting killed... it's time to dig deeper.
-chris
On Sat, Dec 30, 2023 at 12:44 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:
Brian,
On 12/29/23 20:48, Brian Braun wrote:
Hello,
First of all:
Christopher Schultz: You answered an email from me 6 weeks ago. You
helped
me a lot with your suggestions. I have done a lot of research and have
learnt a lot since then, so I have been able to rule out a lot of
potential
roots for my issue. Because of that I am able to post a new more
specific
email. Thanks a lot!!!
Now, this is my stack:
- Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been
enough
for years.
- Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1
2023-08-24
- Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx1000m
-Xms1000m
......")
- My app, which I developed myself, and has been running without any
problems for years
Well, a couple of months ago my website/Tomcat/Java started eating more
and
more memory about after about 4-7 days. The previous days it uses just
a
few hundred MB and is very steady, but then after a few days the memory
usage suddenly grows up to 1.5GB (and then stops growing at that point,
which is interesting). Between these anomalies the RAM usage is fine
and
very steady (as it has been for years) and it uses just about 40-50% of
the
"Max memory" (according to what the Tomcat Manager server status
shows).
The 3 components of G1GC heap memory are steady and low, before and
after
the usage grows to 1.5GB, so it is definitely not that the heap starts
requiring more and more memory. I have been using several tools to
monitor
that (New Relic, VisualVM and JDK Mission Control) so I'm sure that the
memory usage by the heap is not the problem.
The Non-heaps memory usage is not the problem either. Everything there
is
normal, the usage is humble and even more steady.
And there are no leaks, I'm sure of that. I have inspected the JVM
using
several tools.
There are no peaks in the number of threads either. The peak is the
same
when the memory usage is low and when it requires 1.5GB. It stays the
same
all the time.
I have also reviewed all the scheduled tasks in my app and lowered the
amount of objects they create, which was nice and entertaining. But
that
is
not the problem, I have analyzed the object creation by all the threads
(and there are many) and the threads created by my scheduled tasks are
very
humble in their memory usage, compared to many other threads.
And I haven't made any relevant changes to my app in the 6-12 months
before
this problem started occurring. It is weird that I started having this
problem. Could it be that I received an update in the java version or
the
Tomcat version that is causing this problem?
If neither the heap memory or the Non-heaps memory is the source of the
growth of the memory usage, what could it be? Clearly something is
happening inside the JVM that raises the memory usage. And everytime it
grows, it doesn't decrease. It is like if something suddenly starts
"pushing" the memory usage more and more, until it stops at 1.5GB.
I think that maybe the source of the problem is the garbage collector.
I
haven't used any of the switches that we can use to optimize that,
basically because I don't know what I should do there (if I should at
all).
I have also activated the GC log, but I don't know how to analyze it.
I have also increased and decreased the value of "-Xms" parameter and
it
is
useless.
Finally, maybe I should add that I activated 4GB of SWAP memory in my
Ubuntu instance so at least my JVM would not be killed my the OS
anymore
(since the real memory is just 1.8GB). That worked and now the memory
usage
can grow up to 1.5GB without crashing, by using the much slower SWAP
memory, but I still think that this is an abnormal situation.
Thanks in advance for your suggestions!
First of all: what is the problem? Are you just worried that the number
of bytes taken by your JVM process is larger than it was ... sometime in
the past? Or are you experiencing Java OOME of Linux oom-killer or
anything like that?
Not all JVMs behave this way, most most of them do: once memory is
"appropriated" by the JVM from the OS, it will never be released. It's
just too expensive of an operation to shrink the heap.. plus, you told
the JVM "feel free to use up to 1GiB of heap" so it's taking you at your
word. Obviously, the native heap plus stack space for every thread plus
native memory for any native libraries takes up more space than just the
1GiB you gave for the heap, so ... things just take up space.
Lowering the -Xms will never reduce the maximum memory the JVM ever
uses. Only lowering -Xmx can do that. I always recommend setting Xms ==
Xmx because otherwise you are lying to yourself about your needs.
You say you've been running this application "for years". Has it been in
a static environment, or have you been doing things such as upgrading
Java and/or Tomcat during that time? There are things that Tomcat does
now that it did not do in the past that sometimes require more memory to
manage, sometimes only at startup and sometimes for the lifetime of the
server. There are some things that the JVM is doing that require more
memory than their previous versions.
And then there is the usage of your web application. Do you have the
same number of users? I've told this (short)( story a few times on this
list, but we had a web application that ran for 10 years with only 64MiB
of heap and one day we started getting OOMEs. At first we just bounced
the service and tried looking for bugs, leaks, etc. but the heap dumps
were telling us everything was fine.
The problem was user load. We simply outgrew the heap we had allocated
because we had more simultaneous logged-in users than we did in the
past, and they all had sessions, etc. We had plenty of RAM available, we
were just being stingy with it.
The G1 garbage collector doesn't have very many switches to mess-around
with it compared to older collectors. The whole point of G1 was to "make
garbage collection easy". Feel free to read 30 years of lies and
confusion about how to best configure Java garbage collectors. At the
end of the day, if you don't know exactly what you are doing and/or you
don't have a specific problem you are trying to solve, you are better
off leaving everything with default settings.
If you want to reduce the amount of RAM your application uses, set a
lower heap size. If that causes OOMEs, audit your application for wasted
memory such as too-large caches (which presumably live a long time) or
too-large single-transactions such as loading 10k records all at once
from a database. Sometimes a single request can require a whole lot of
memory RIGHT NOW which is only used temporarily.
I was tracking-down something in our own application like this recently:
a page-generation process was causing an OOME periodically, but the JVM
was otherwise very healthy. It turns out we had an administrative action
in our application that had no limits on the amount of data that could
be requested from the database at once. So naive administrators were
able to essentially cause a query to be run that returned a huge number
of rows from the db, then every row was being converted into a row in an
HTML table in a web page. Our page-generation process builds the whole
page in memory before returning it, instead of streaming it back out to
the user, which means a single request can use many MiBs of memory just
for in-memory strings/byte arrays.
If something like that happens in your application, it can pressure the
heap to jump from e.g. 256MiB way up to 1.5GiB and -- as I said before
-- the JVM is never gonna give that memory back to the OS.
So even though everything "looks good", your heap and native memory
spaces are very large until you terminate the JVM.
If you haven't already done so, I would recommend that you enable GC
logging. How to do that is very dependent on your JVM, version, and
environment. This writes GC activity details to a series of files during
the JVM execution. There are freely-available tools you can use to view
those log files in a meaningful way and draw some conclusions. You might
even be able to see when that "memory event" took place that caused your
heap memory to shoot-up. (Or maybe it's your native memory, which isn't
logged by the GC logger.) If you are able to see when it happened, you
may be able to correlate that with your application log to see what
happened in your application. Maybe you need a fix.
Then again, maybe everything is totally fine and there is nothing to
worry about.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
