Hi Chris, Thanks a lot for your very detailed response! Here are my answers, comments and questions.
At the beginning, this was the problem: The OOM-killer (something that I never knew existed) killing Tomcat unexpectedly and without any explanation, many times during the night while I should be sleeping peacefully (as long as I wear my Apnea mask). Not knowing how much memory would I need to satisfy the JVM, and not willing to migrate to more expensive Amazon instances just because I don't know why this is happening. And not knowing if the memory requirement would keep growing and growing and growing. Then I activated the SWAP file, and I discovered that this problem stops at 1.5GB of memory used by the JVM. At least I am not getting more crashes anymore. But I consider the SWAP file as a palliative and I really want to know what is the root of this problem. If I don't, then maybe I should consider another career. I don't enjoy giving up. Yes, the memory used by the JVM started to grow suddenly one day, after several years running fine. Since I had not made any changes to my app, I really don't know the reason. And I really think this should not be happening without an explanation. I don't have any Java OOME exceptions, so it is not that my objects don't fit. Even if I supply 300MB to the -Xmx parameter. In fact, as I wrote, I don't think the Heap and non-heap usage is the problem. I have been inspecting those and their usage seems to be normal/modest and steady. I can see that using the Tomcat Manager as well as several other tools (New Relic, VisualVM, etc). Regarding the 1GB I am giving now to the -Xms parameter: I was giving just a few hundreds and I already had the problem. Actually I think it is the same if I give a few hundreds of MBs or 1GB, the JVM still starts using more memory after 3-4 days of running until it takes 1.5GB. But during the first 1-4 days it uses just a few hundred MBs. My app has been "static" as you say, but probably I have upgraded Tomcat and/or Java recently. I don't really remember. Maybe one of those upgrades brought this issue as a result. Actually, If I knew that one of those upgrades causes this huge pike in memory consumption and there is no way to avoid it, then I would accept it as a fact of life and move on. But since I don't know, it really bugs me. I have the same amount of users and traffic as before. I also know how much memory a session takes and it is fine. I have also checked the HTTP(S) requests to see if somehow I am getting any attempts to hack my instance that could be the root of this problem. Yes, I get hacking attempts by those bots all the time, but I don't see anything relevant there. No news. I agree with what you say now regarding the GC. I should not need to use those switches since I understand it should work fine without using them. And I don't know how to use them. And since I have never cared about using them for about 15 years using Java+Tomcat, why should I start now? I have also checked all my long-lasting objects. I have optimized my DB queries recently as you suggest now, so they don't create huge amounts of objects in a short period of time that the GC would have to deal with. The same applies to my scheduled tasks. They all run very quickly and use modest amounts of memory. All the other default Tomcat threads create far more objects. I have already activated the GC log. Is there a tool that you would suggest to analyze it? I haven't even opened it. I suspect that the root of my problem comes from the GC process indeed. Thanks again! Brian On Sat, Dec 30, 2023 at 12:44 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > Brian, > > On 12/29/23 20:48, Brian Braun wrote: > > Hello, > > > > First of all: > > Christopher Schultz: You answered an email from me 6 weeks ago. You > helped > > me a lot with your suggestions. I have done a lot of research and have > > learnt a lot since then, so I have been able to rule out a lot of > potential > > roots for my issue. Because of that I am able to post a new more specific > > email. Thanks a lot!!! > > > > Now, this is my stack: > > > > - Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been enough > > for years. > > - Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1 > 2023-08-24 > > - Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx1000m -Xms1000m > > ......") > > - My app, which I developed myself, and has been running without any > > problems for years > > > > Well, a couple of months ago my website/Tomcat/Java started eating more > and > > more memory about after about 4-7 days. The previous days it uses just a > > few hundred MB and is very steady, but then after a few days the memory > > usage suddenly grows up to 1.5GB (and then stops growing at that point, > > which is interesting). Between these anomalies the RAM usage is fine and > > very steady (as it has been for years) and it uses just about 40-50% of > the > > "Max memory" (according to what the Tomcat Manager server status shows). > > The 3 components of G1GC heap memory are steady and low, before and after > > the usage grows to 1.5GB, so it is definitely not that the heap starts > > requiring more and more memory. I have been using several tools to > monitor > > that (New Relic, VisualVM and JDK Mission Control) so I'm sure that the > > memory usage by the heap is not the problem. > > The Non-heaps memory usage is not the problem either. Everything there is > > normal, the usage is humble and even more steady. > > > > And there are no leaks, I'm sure of that. I have inspected the JVM using > > several tools. > > > > There are no peaks in the number of threads either. The peak is the same > > when the memory usage is low and when it requires 1.5GB. It stays the > same > > all the time. > > > > I have also reviewed all the scheduled tasks in my app and lowered the > > amount of objects they create, which was nice and entertaining. But that > is > > not the problem, I have analyzed the object creation by all the threads > > (and there are many) and the threads created by my scheduled tasks are > very > > humble in their memory usage, compared to many other threads. > > > > And I haven't made any relevant changes to my app in the 6-12 months > before > > this problem started occurring. It is weird that I started having this > > problem. Could it be that I received an update in the java version or the > > Tomcat version that is causing this problem? > > > > If neither the heap memory or the Non-heaps memory is the source of the > > growth of the memory usage, what could it be? Clearly something is > > happening inside the JVM that raises the memory usage. And everytime it > > grows, it doesn't decrease. It is like if something suddenly starts > > "pushing" the memory usage more and more, until it stops at 1.5GB. > > > > I think that maybe the source of the problem is the garbage collector. I > > haven't used any of the switches that we can use to optimize that, > > basically because I don't know what I should do there (if I should at > all). > > I have also activated the GC log, but I don't know how to analyze it. > > > > I have also increased and decreased the value of "-Xms" parameter and it > is > > useless. > > > > Finally, maybe I should add that I activated 4GB of SWAP memory in my > > Ubuntu instance so at least my JVM would not be killed my the OS anymore > > (since the real memory is just 1.8GB). That worked and now the memory > usage > > can grow up to 1.5GB without crashing, by using the much slower SWAP > > memory, but I still think that this is an abnormal situation. > > > > Thanks in advance for your suggestions! > > First of all: what is the problem? Are you just worried that the number > of bytes taken by your JVM process is larger than it was ... sometime in > the past? Or are you experiencing Java OOME of Linux oom-killer or > anything like that? > > Not all JVMs behave this way, most most of them do: once memory is > "appropriated" by the JVM from the OS, it will never be released. It's > just too expensive of an operation to shrink the heap.. plus, you told > the JVM "feel free to use up to 1GiB of heap" so it's taking you at your > word. Obviously, the native heap plus stack space for every thread plus > native memory for any native libraries takes up more space than just the > 1GiB you gave for the heap, so ... things just take up space. > > Lowering the -Xms will never reduce the maximum memory the JVM ever > uses. Only lowering -Xmx can do that. I always recommend setting Xms == > Xmx because otherwise you are lying to yourself about your needs. > > You say you've been running this application "for years". Has it been in > a static environment, or have you been doing things such as upgrading > Java and/or Tomcat during that time? There are things that Tomcat does > now that it did not do in the past that sometimes require more memory to > manage, sometimes only at startup and sometimes for the lifetime of the > server. There are some things that the JVM is doing that require more > memory than their previous versions. > > And then there is the usage of your web application. Do you have the > same number of users? I've told this (short)( story a few times on this > list, but we had a web application that ran for 10 years with only 64MiB > of heap and one day we started getting OOMEs. At first we just bounced > the service and tried looking for bugs, leaks, etc. but the heap dumps > were telling us everything was fine. > > The problem was user load. We simply outgrew the heap we had allocated > because we had more simultaneous logged-in users than we did in the > past, and they all had sessions, etc. We had plenty of RAM available, we > were just being stingy with it. > > The G1 garbage collector doesn't have very many switches to mess-around > with it compared to older collectors. The whole point of G1 was to "make > garbage collection easy". Feel free to read 30 years of lies and > confusion about how to best configure Java garbage collectors. At the > end of the day, if you don't know exactly what you are doing and/or you > don't have a specific problem you are trying to solve, you are better > off leaving everything with default settings. > > If you want to reduce the amount of RAM your application uses, set a > lower heap size. If that causes OOMEs, audit your application for wasted > memory such as too-large caches (which presumably live a long time) or > too-large single-transactions such as loading 10k records all at once > from a database. Sometimes a single request can require a whole lot of > memory RIGHT NOW which is only used temporarily. > > I was tracking-down something in our own application like this recently: > a page-generation process was causing an OOME periodically, but the JVM > was otherwise very healthy. It turns out we had an administrative action > in our application that had no limits on the amount of data that could > be requested from the database at once. So naive administrators were > able to essentially cause a query to be run that returned a huge number > of rows from the db, then every row was being converted into a row in an > HTML table in a web page. Our page-generation process builds the whole > page in memory before returning it, instead of streaming it back out to > the user, which means a single request can use many MiBs of memory just > for in-memory strings/byte arrays. > > If something like that happens in your application, it can pressure the > heap to jump from e.g. 256MiB way up to 1.5GiB and -- as I said before > -- the JVM is never gonna give that memory back to the OS. > > So even though everything "looks good", your heap and native memory > spaces are very large until you terminate the JVM. > > If you haven't already done so, I would recommend that you enable GC > logging. How to do that is very dependent on your JVM, version, and > environment. This writes GC activity details to a series of files during > the JVM execution. There are freely-available tools you can use to view > those log files in a meaningful way and draw some conclusions. You might > even be able to see when that "memory event" took place that caused your > heap memory to shoot-up. (Or maybe it's your native memory, which isn't > logged by the GC logger.) If you are able to see when it happened, you > may be able to correlate that with your application log to see what > happened in your application. Maybe you need a fix. > > Then again, maybe everything is totally fine and there is nothing to > worry about. > > -chris > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >