Re: Tomcat/Java starts using too much memory and not by the heap or non-heap memory
Hi Brian, Am 30.12.2023 um 21:42 schrieb Brian Braun: I don't have any Java OOME exceptions, so it is not that my objects don't fit. Even if I supply 300MB to the -Xmx parameter. In fact, as I wrote, I don't think the Heap and non-heap usage is the problem. I have been inspecting those and their usage seems to be normal/modest and steady. I can see that using the Tomcat Manager as well as several other tools (New Relic, VisualVM, etc). Regarding the 1GB I am giving now to the -Xms parameter: I was giving just a few hundreds and I already had the problem. Actually I think it is the same if I give a few hundreds of MBs or 1GB, the JVM still starts using more memory after 3-4 days of running until it takes 1.5GB. But during the first 1-4 days it uses just a few hundred MBs. So if this is not heap memory (-Xmx) it must be some other memory the JVM uses. I guess we can rule out reserved code cache (-XX:ReservedCodeCacheSize) and stack size (-Xss) because they should have fixed sizes and you've written earlier that you checked the number of threads. This leaves us with meta space (-XX:MaxMetaspaceSize) and native/direct memory (-XX:MaxDirectMemorySize). You can try to limit that or use the Java flight recorder and tools like Mission Control or VisualVM to make that kind of memory usage visible. Regards, Stefan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat/Java starts using too much memory and not by the heap or non-heap memory
Hi Chris, Thanks a lot for your very detailed response! Here are my answers, comments and questions. At the beginning, this was the problem: The OOM-killer (something that I never knew existed) killing Tomcat unexpectedly and without any explanation, many times during the night while I should be sleeping peacefully (as long as I wear my Apnea mask). Not knowing how much memory would I need to satisfy the JVM, and not willing to migrate to more expensive Amazon instances just because I don't know why this is happening. And not knowing if the memory requirement would keep growing and growing and growing. Then I activated the SWAP file, and I discovered that this problem stops at 1.5GB of memory used by the JVM. At least I am not getting more crashes anymore. But I consider the SWAP file as a palliative and I really want to know what is the root of this problem. If I don't, then maybe I should consider another career. I don't enjoy giving up. Yes, the memory used by the JVM started to grow suddenly one day, after several years running fine. Since I had not made any changes to my app, I really don't know the reason. And I really think this should not be happening without an explanation. I don't have any Java OOME exceptions, so it is not that my objects don't fit. Even if I supply 300MB to the -Xmx parameter. In fact, as I wrote, I don't think the Heap and non-heap usage is the problem. I have been inspecting those and their usage seems to be normal/modest and steady. I can see that using the Tomcat Manager as well as several other tools (New Relic, VisualVM, etc). Regarding the 1GB I am giving now to the -Xms parameter: I was giving just a few hundreds and I already had the problem. Actually I think it is the same if I give a few hundreds of MBs or 1GB, the JVM still starts using more memory after 3-4 days of running until it takes 1.5GB. But during the first 1-4 days it uses just a few hundred MBs. My app has been "static" as you say, but probably I have upgraded Tomcat and/or Java recently. I don't really remember. Maybe one of those upgrades brought this issue as a result. Actually, If I knew that one of those upgrades causes this huge pike in memory consumption and there is no way to avoid it, then I would accept it as a fact of life and move on. But since I don't know, it really bugs me. I have the same amount of users and traffic as before. I also know how much memory a session takes and it is fine. I have also checked the HTTP(S) requests to see if somehow I am getting any attempts to hack my instance that could be the root of this problem. Yes, I get hacking attempts by those bots all the time, but I don't see anything relevant there. No news. I agree with what you say now regarding the GC. I should not need to use those switches since I understand it should work fine without using them. And I don't know how to use them. And since I have never cared about using them for about 15 years using Java+Tomcat, why should I start now? I have also checked all my long-lasting objects. I have optimized my DB queries recently as you suggest now, so they don't create huge amounts of objects in a short period of time that the GC would have to deal with. The same applies to my scheduled tasks. They all run very quickly and use modest amounts of memory. All the other default Tomcat threads create far more objects. I have already activated the GC log. Is there a tool that you would suggest to analyze it? I haven't even opened it. I suspect that the root of my problem comes from the GC process indeed. Thanks again! Brian On Sat, Dec 30, 2023 at 12:44 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > Brian, > > On 12/29/23 20:48, Brian Braun wrote: > > Hello, > > > > First of all: > > Christopher Schultz: You answered an email from me 6 weeks ago. You > helped > > me a lot with your suggestions. I have done a lot of research and have > > learnt a lot since then, so I have been able to rule out a lot of > potential > > roots for my issue. Because of that I am able to post a new more specific > > email. Thanks a lot!!! > > > > Now, this is my stack: > > > > - Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been enough > > for years. > > - Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1 > 2023-08-24 > > - Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx1000m -Xms1000m > > ..") > > - My app, which I developed myself, and has been running without any > > problems for years > > > > Well, a couple of months ago my website/Tomcat/Java started eating more > and > > more memory about after about 4-7 days. The previous days it uses just a > > few hundred MB and is very steady, but then after a few days the memory > > usage suddenly grows up to 1.5GB (and then stops growing at that point, > > which is interesting). Between these anomalies the RAM usage is fine and > > very steady (as it has been for years) and it uses just about 40-50% of > the > > "Max memory"
Re: Tomcat/Java starts using too much memory and not by the heap or non-heap memory
Brian, On 12/29/23 20:48, Brian Braun wrote: Hello, First of all: Christopher Schultz: You answered an email from me 6 weeks ago. You helped me a lot with your suggestions. I have done a lot of research and have learnt a lot since then, so I have been able to rule out a lot of potential roots for my issue. Because of that I am able to post a new more specific email. Thanks a lot!!! Now, this is my stack: - Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been enough for years. - Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1 2023-08-24 - Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx1000m -Xms1000m ..") - My app, which I developed myself, and has been running without any problems for years Well, a couple of months ago my website/Tomcat/Java started eating more and more memory about after about 4-7 days. The previous days it uses just a few hundred MB and is very steady, but then after a few days the memory usage suddenly grows up to 1.5GB (and then stops growing at that point, which is interesting). Between these anomalies the RAM usage is fine and very steady (as it has been for years) and it uses just about 40-50% of the "Max memory" (according to what the Tomcat Manager server status shows). The 3 components of G1GC heap memory are steady and low, before and after the usage grows to 1.5GB, so it is definitely not that the heap starts requiring more and more memory. I have been using several tools to monitor that (New Relic, VisualVM and JDK Mission Control) so I'm sure that the memory usage by the heap is not the problem. The Non-heaps memory usage is not the problem either. Everything there is normal, the usage is humble and even more steady. And there are no leaks, I'm sure of that. I have inspected the JVM using several tools. There are no peaks in the number of threads either. The peak is the same when the memory usage is low and when it requires 1.5GB. It stays the same all the time. I have also reviewed all the scheduled tasks in my app and lowered the amount of objects they create, which was nice and entertaining. But that is not the problem, I have analyzed the object creation by all the threads (and there are many) and the threads created by my scheduled tasks are very humble in their memory usage, compared to many other threads. And I haven't made any relevant changes to my app in the 6-12 months before this problem started occurring. It is weird that I started having this problem. Could it be that I received an update in the java version or the Tomcat version that is causing this problem? If neither the heap memory or the Non-heaps memory is the source of the growth of the memory usage, what could it be? Clearly something is happening inside the JVM that raises the memory usage. And everytime it grows, it doesn't decrease. It is like if something suddenly starts "pushing" the memory usage more and more, until it stops at 1.5GB. I think that maybe the source of the problem is the garbage collector. I haven't used any of the switches that we can use to optimize that, basically because I don't know what I should do there (if I should at all). I have also activated the GC log, but I don't know how to analyze it. I have also increased and decreased the value of "-Xms" parameter and it is useless. Finally, maybe I should add that I activated 4GB of SWAP memory in my Ubuntu instance so at least my JVM would not be killed my the OS anymore (since the real memory is just 1.8GB). That worked and now the memory usage can grow up to 1.5GB without crashing, by using the much slower SWAP memory, but I still think that this is an abnormal situation. Thanks in advance for your suggestions! First of all: what is the problem? Are you just worried that the number of bytes taken by your JVM process is larger than it was ... sometime in the past? Or are you experiencing Java OOME of Linux oom-killer or anything like that? Not all JVMs behave this way, most most of them do: once memory is "appropriated" by the JVM from the OS, it will never be released. It's just too expensive of an operation to shrink the heap.. plus, you told the JVM "feel free to use up to 1GiB of heap" so it's taking you at your word. Obviously, the native heap plus stack space for every thread plus native memory for any native libraries takes up more space than just the 1GiB you gave for the heap, so ... things just take up space. Lowering the -Xms will never reduce the maximum memory the JVM ever uses. Only lowering -Xmx can do that. I always recommend setting Xms == Xmx because otherwise you are lying to yourself about your needs. You say you've been running this application "for years". Has it been in a static environment, or have you been doing things such as upgrading Java and/or Tomcat during that time? There are things that Tomcat does now that it did not do in the past that sometimes require more memory to manage, sometimes only at startup and sometimes
Re: Aw: Re: Servlet-Mapping having %-sign
Peter, On 12/30/23 01:41, Peter Rader wrote: Peter, On 12/29/23 07:56, Peter Rader wrote: having a URL like this: https://localhost:8443/index.html works perfect. This is my mapping: Nano-Nano-Servlet /index.html Nano-Nano-Servlet *.ts Unfortunately this URI does not load (because of the %-sign): https://localhost:8443/@rm%2fmodel.ts It gives a http-status:400 having the message "Invalid URI: [noSlash]" What's the use-case for having a client use a %-encoded / in your URL? That kind of thing is usually evidence of a hacking attempt, which is why Tomcat returns a 400 response. I generate TypeScript dynamically. In order to use it in Node: I register a servlet to create npm-packages at run-time. On Node-side I use this command: 1. Register servlet as npm source: 'npm config set @myapp:registry=https://nonofyourbusiness.mydomain.com:8443/' 2. Start the download: 'npm install @myapp/model --loglevel verbose' (hint: @myapp is the tomcat) This is the output of the second command: npm info it worked if it ends with ok npm verb cli [ npm verb cli '/home/grim/.nvm/versions/node/v14.18.1/bin/node', npm verb cli '/home/grim/.nvm/versions/node/v14.18.1/bin/npm', npm verb cli 'install', npm verb cli '@myapp/model@1.0.0', npm verb cli '--loglevel', npm verb cli 'verbose' npm verb cli ] npm info using npm@6.14.15 npm info using node@v14.18.1 npm verb config Skipping project config: /home/grim/.npmrc. (matches userconfig) npm verb npm-session 778f7308eede99d8 npm http fetch GET 200 https://nonofyourbusiness.mydomain.com:8443/@myapp%2fmodel 28ms npm http fetch GET 200 https://nonofyourbusiness.mydomain.com:8443/index.tgz.ts 14ms npm timing stage:loadCurrentTree Completed in 71ms npm timing stage:loadIdealTree:cloneCurrentTree Completed in 0ms npm timing stage:loadIdealTree:loadShrinkwrap Completed in 3ms npm timing stage:loadIdealTree:loadAllDepsIntoIdealTree Completed in 1ms npm timing stage:loadIdealTree Completed in 5ms npm timing stage:generateActionsToTake Completed in 1ms npm verb correctMkdir /home/grim/.npm/_locks correctMkdir not in flight; initializing npm verb lock using /home/grim/.npm/_locks/staging-b24acfc1530c2325.lock for /home/grim/node_modules/.staging npm http fetch GET 200 https://nonofyourbusiness.mydomain.com:8443/index.tgz.ts 7ms npm timing action:extract Completed in 10ms npm timing action:finalize Completed in 1ms npm timing action:refresh-package-json Completed in 1ms npm info lifecycle model@1.0.0~preinstall: model@1.0.0 npm timing action:preinstall Completed in 1ms npm info linkStuff model@1.0.0 npm timing action:build Completed in 0ms npm info lifecycle model@1.0.0~install: model@1.0.0 npm timing action:install Completed in 1ms npm info lifecycle model@1.0.0~postinstall: model@1.0.0 npm timing action:postinstall Completed in 0ms npm verb unlock done using /home/grim/.npm/_locks/staging-b24acfc1530c2325.lock for /home/grim/node_modules/.staging npm timing stage:executeActions Completed in 18ms npm timing stage:rollbackFailedOptional Completed in 1ms npm timing stage:runTopLevelLifecycles Completed in 97ms npm WARN saveError ENOENT: no such file or directory, open '/home/grim/package.json' npm info lifecycle undefined~preshrinkwrap: undefined npm info lifecycle undefined~shrinkwrap: undefined npm info lifecycle undefined~postshrinkwrap: undefined npm WARN enoent ENOENT: no such file or directory, open '/home/grim/package.json' npm verb enoent This is related to npm not being able to find a file. npm verb enoent npm WARN grim No description npm WARN grim No repository field. npm WARN grim No README data npm WARN grim No license field. npm http fetch POST 400 https://registry.npmjs.org/-/npm/v1/security/audits/quick 266ms + model@1.0.0 (as @myapp/model) added 1 package in 0.347s npm verb exit [ 0, true ] npm timing npm Completed in 463ms npm info ok --- end of console output As you might have noticed, this time the URL responded successfully. This is because I modified catalina.properties (org.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true). https://stackoverflow.com/questions/19576777/why-does-apache-tomcat-handle-encoded-slashes-2f-as-path-separators I agree that this might become a security risk. Since the mentioned mod_jk-bug is not affected in this particular case, I could exoticize the tomcat-config to undo tomcats built-in-workaround throu the catalina.properties. It does not feel like an elegant solution, but it works for now. If however npm might be the future for some developers, a redesign of tomcat may a more desirable solution. It might be hard to tell the npm people to change their "way of downloading npm-packages" because "mod_jk have a bug" might not be concidered as a convincing argument. :-D This is not just a "mod_jk had a bug one time" issue, but more of a design decision in Tomcat of where to perform the decoding and normalization of the URL. Changing the way Tomcat behaves could have lots of
Re: Admin password for Tomcat
Jerry,
On 12/30/23 01:20, Jerry Malcolm wrote:
Chris,
On 12/29/2023 11:22 AM, Christopher Schultz wrote:
Jerry,
On 12/28/23 18:33, Jerry Malcolm wrote:
Chris,
On 12/28/2023 3:38 PM, Christopher Schultz wrote:
Jerry.
On 12/27/23 02:13, Jerry Malcolm wrote:
I implemented the filter as you suggested. But I guess I'm going
to need some education on sessions. Down in a user profile web
page I have a button to "Impersonate".
I'm with you so far.
I create the GenericPrincipal object and store it in the session.
I've checked several times, and every time I come back to that
code, the attribute is set in the session object.
Good. When you do that, do you remove the "real" user's
GenericPrincipal object from the session? Or are they both in there?
>
Sorry... lost you on that one. I am just setting a custom >
"GenericPrincipal" attribute named "impersonatedPrincipal" in the
session when a user clicks the "Impersonate" button on the web page.
This answers my question. I was wondering if you do this:
session.setAttribute("user", impersonatedUser); // Replace
or this:
session.setAttribut("impersonatedUser", impersonatedUser);
And it seems you are doing the second one.
In my understanding, at this point I'm just 'telling' the session
that on subsequent requests in the custom filter, here's a principal
object that I want to insert.
As long as your code agrees with you :)
I also noted in your early example that you stored a 'User' class as
the attribute in the session, not a GenericPrincipal. I couldn't
find a "User" class. So I just used GenericPrincipal, since that was
what I will insert in the request object in the filter.
We use User in our session, and essentially wrap it in a
GenericPrincipal when necessary. We are playing a lot of games, here,
in our code, so I apologize if we go down this road and it's a lot
longer than you had expected...
Remember that if Tomcat is going to enforce your authentication and
authorization constraints, your Filter will run after that, and
Tomcat and your application will disagree over which user is
currently logged-in.
>
I'm not removing the real principal from anything. Not sure how to
do that? in HttpSession? in HttpRequest? I assumed returning my
new GenericPrincipal in the RequestFacade would override any other
code asking for the principal.
It will... unless that code runs before your Filter has a chance to
pull the wool over the application's eyes. For example... Tomcat's
authentication and authorization code will run in a Valve, which runs
before all Filters.
How do I go about removing the real principal?
Let's save that for later ;)
But when I put breakpoint in my new Filter object and look in the
session, no attribute. It's a different session object from what I
can tell.
That's weird.
When you say "every time I come back to that code, the attribute is
set in the session" ... what code are you taking about?
>
The filter's 'version' of the session doesn't have the
"impersonatedPrincipal" attribute set (it doesn't have any attributes
set). But after clicking Impersonate, hitting the breakpoint, and
watching the session attribute get set, I hit F5 to refresh the page.
The filter breakpoint again doesn't have the attribute. But if I
click "Impersonate" again and hit that breakpoint the
"impersonatedPrincipal" session attribute exists in the session.
Is the session identifier changing?
I really thought I understood session objects. I thought there was
only one session object throughout the processing of a servlet.
Yes, if s/servlet/request/.
But I'm obviously missing something in the flows. Why is there a
different session object in the filter than in the main body of the
servlet? I did the getSession(false) as you suggested. The session
object is not null. It just doesn't have the attribute set. Yet if
I hit the Impersonate button again and hit the breakpoint, the
GenericPrincipal attribute is sitting in the session just as I placed
it earlier.
If the difference between when Tomcat evaluates e.g. user-roles
versus when your application does won't explain what's happening, we
might need to see some code.
Code:
Other than a loop that builds a Roles vector, these are the two lines
that create the session attribute when the "Impersonate" button is
clicked.
GenericPrincipal newPrincipal = new GenericPrincipal( getUserName(),
getPassword(),
roles );
getCtrl().getRequest().getSession(false).setAttribute(
"impersonatedPrincipal", newPrincipal );
Hmm. What are Ctrl and Request that you are "getting"? Usually, both
the servlet and the Filter see objects passed to them directly in the
doFilter() and service()/doGet/doPost/whatever methods in the servlet.
And this is the filter:
PrintWriter out = response.getWriter();
HttpSession session =
((HttpServletRequest)request).getSession(false);
if(session != null)
{
final
