RE: Very Long Full GC after Inactivity
From: Bill Clarke-Fields [mailto:[EMAIL PROTECTED] If it turns out to be so, is there anything I can do to force it to not page out the JVM heap? There's cheap and there's messy :-). The cheap way: Set up a job that runs overnight that exercises your app. It should keep the critical files paged in. The messy way: Run Tomcat as an admin (ick!) and make a JNI call to some native code that pins Tomcat's address space into RAM. I have not tried this, don't know whether there's a correct JVM way of doing it, and have *no* idea how it might interact with Sun's address space management! It's something to try very, very cautiously - this might bite you in unexpected ways. The cheap *and* messy way: Use Task Manager to boost Tomcat's priority to realtime. I *think* that pins the process' address space into RAM, although check as I'm stating this from memory and my memory is erratic these days. Things will get very unpleasant if Tomcat manages to take 100% CPU on all your cores at once, as a Realtime process is running at higher priority than the UI so you'd have no way to interrupt it. On modern multi-core processors, that's less of an issue - set the Affinity mask for Tomcat *before* raising the priority so there's at least one core it's not allowed to run on, and you'll be able to regain control. As I say, not sure this will work, but it's an interesting tactic to try :-). Side note: Does your app also use a database? On which machine? Is your database server also being paged out, adding to the slowness? If it's SQL Server, it'll be requested to drop cache pages instead as the system comes under memory pressure, so the heap will quietly shrink... - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Very Long Full GC after Inactivity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Peter, Peter Crowther wrote: The messy way: Run Tomcat as an admin (ick!) and make a JNI call to some native code that pins Tomcat's address space into RAM. Messy indeed. I have not tried this, don't know whether there's a correct JVM way of doing it, and have *no* idea how it might interact with Sun's address space management! It's something to try very, very cautiously - this might bite you in unexpected ways. My experience with JNI is that you have to pin objects one at a time. Since Tomcat manages millions of objects, I'm not sure how you'd do this effectively. Also, I believe that pinning an object does more than make is non-relocatable within the heap (see below); I think that if you pinned objects indefinitely, Bad Things would probably happen. One Bad Thing that will probably happen is that objects will never be tenured, and you'll fill up other parts of your heap with objects that ought to be promoted to more long-lasting areas of the heap. Finally, pinning an object has no bearing on its pagability... it just means that a native pointer can be used to access it, and the memory address where the object can be found won't change until it's unpinned. The memory can still be swapped to disk, as long as the memory address for that object does not change. Heck, it can even be relocate in physical memory of the OS wants to do it (which it probably will do during paging), as long as the process can always use that same pointer to access the object. I'd imagine that most OSs have functions to /actually/ pin data into memory (making is non-pagable). In order to do that, you'd have to Java-pin the object, then OS-pin the memory. I think that's going to cause lots more problems than a long wait for the first request of the day ;) I actually favor your cheap way since it's super simple and likely to work just as well ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHDN9X9CaO5/Lv0PARAhPhAKCfmYrgUs1/4dhdkDtC8k26K32CdQCfZ56H OPkbbsDv9mtnMGRDMejdjFw= =cOdy -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
From: Christopher Schultz [mailto:[EMAIL PROTECTED] My experience with JNI is that you have to pin objects one at a time. To be clear: I wasn't talking about pinning Java objects to particular addresses in memory, instead pinning the entire process address space into memory using Win32 API calls. I'd imagine that most OSs have functions to /actually/ pin data into memory (making is non-pagable). Exactly. In order to do that, you'd have to Java-pin the object, then OS-pin the memory. Why Java-pin anything? (assuming you use OS-pin the entire Java heap, which I accept may be a non-trivial exercise) I actually favor your cheap way since it's super simple and likely to work just as well ;) Well, me too - it involves the least disruption. KISS says it's the one to go for if it can be made to work. - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: Very Long Full GC after Inactivity My experience with JNI is that you have to pin objects one at a time. Peter's not talking about the JNI pin function, which only prevents objects from moving around in the JVM heap, not in RAM. He's referring to nailing the entire heap into RAM, which is an OS call requiring privileges; operations of GC within the JVM would be unaffected. It would be a simple thing to do, but would really foul up the workings of the OS memory manager. (Been there, done that, strongly discourage it.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
From: Bill Clarke-Fields [mailto:[EMAIL PROTECTED] We are running an application on Tomcat 5.0.28 with Java 1.4.2. The usage of the application is very cyclical. It is used heavily during the day, and lightly at night. During peak daytime hours, a full garbage collection takes less than 2 seconds, which is fine. However, after a long period of inactivity in the evening, a full garbage collection will be triggered and take a very long time. Sometimes over a minute! I suspect the app is being paged out overnight by other jobs. Overnight virus scans will do this on Windows, for example, as the disk cache expands to fill available memory. You could test this by running Performance Monitor (Windows) or your preferred logging tool (UNIX) overnight in log mode and looking for high page-in and disk I/O values around the time of the full GC. For preference, use the Process counters on Windows, as they'll tell you for sure whether it's the Java process that's causing the paging traffic. - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
Peter, Thanks! This is a very good idea. I know that there are some virus scans and other processes running at night. I will do some monitoring to determine if paging is the problem. If it turns out to be so, is there anything I can do to force it to not page out the JVM heap? Thanks again, -Bill Peter Crowther wrote: From: Bill Clarke-Fields [mailto:[EMAIL PROTECTED] We are running an application on Tomcat 5.0.28 with Java 1.4.2. The usage of the application is very cyclical. It is used heavily during the day, and lightly at night. During peak daytime hours, a full garbage collection takes less than 2 seconds, which is fine. However, after a long period of inactivity in the evening, a full garbage collection will be triggered and take a very long time. Sometimes over a minute! I suspect the app is being paged out overnight by other jobs. Overnight virus scans will do this on Windows, for example, as the disk cache expands to fill available memory. You could test this by running Performance Monitor (Windows) or your preferred logging tool (UNIX) overnight in log mode and looking for high page-in and disk I/O values around the time of the full GC. For preference, use the Process counters on Windows, as they'll tell you for sure whether it's the Java process that's causing the paging traffic. - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Very-Long-Full-GC-after-Inactivity-tf4589459.html#a13118297 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Very Long Full GC after Inactivity
Hi, check the jvm runs in server mode (check there is a '-server' parameter given to jvm startup). Server and client mode have different GC mecanisms. Bill Clarke-Fields a écrit : Hi, We are running an application on Tomcat 5.0.28 with Java 1.4.2. The usage of the application is very cyclical. It is used heavily during the day, and lightly at night. During peak daytime hours, a full garbage collection takes less than 2 seconds, which is fine. However, after a long period of inactivity in the evening, a full garbage collection will be triggered and take a very long time. Sometimes over a minute! Is there any way to avoid this long garbage collection? Why does it take so long? (Maybe because it is filling up with many small objects over time rather than a few larger objects?) Thanks for your help. -Bill p.s. Upgrading to a newer version of Tomcat or Java is unfortunately not an option at this point. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
From: David Delbecq [mailto:[EMAIL PROTECTED] Subject: Re: Very Long Full GC after Inactivity check the jvm runs in server mode (check there is a '-server' parameter given to jvm startup). Server and client mode have different GC mecanisms. That's not correct; the set of GC algorithms available for server and client are identical, but the default heap sizes are different. For initial problem analysis, turn on -verbose:gc if you haven't already, and use JConsole to look at the heap usage. You may need a profiler to find out just what objects are hanging around and causing the long pause. You might want to try running with a later JVM - a 1.5 or 6 version should give you reduced pause times as well as better performance overall. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Very Long Full GC after Inactivity
David, Tomcat is running as a Windows service, and the jvm startup parameter is set to: C:\j2sdk1.4.2_11\jre\bin\server\jvm.dll. Do I also need to pass a -server parameter? Thanks, -Bill David Delbecq-2 wrote: Hi, check the jvm runs in server mode (check there is a '-server' parameter given to jvm startup). Server and client mode have different GC mecanisms. Bill Clarke-Fields a écrit : Hi, We are running an application on Tomcat 5.0.28 with Java 1.4.2. The usage of the application is very cyclical. It is used heavily during the day, and lightly at night. During peak daytime hours, a full garbage collection takes less than 2 seconds, which is fine. However, after a long period of inactivity in the evening, a full garbage collection will be triggered and take a very long time. Sometimes over a minute! Is there any way to avoid this long garbage collection? Why does it take so long? (Maybe because it is filling up with many small objects over time rather than a few larger objects?) Thanks for your help. -Bill p.s. Upgrading to a newer version of Tomcat or Java is unfortunately not an option at this point. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Very-Long-Full-GC-after-Inactivity-tf4589459.html#a13101254 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
Charles, I have tuned the heap size based on what is ideal during peak hours. The problem is that off-hours usage is very different, and leads to these very long full GCs. Does it sound like an application problem? I have not done any analysis to determine what kind of objects are hanging around. This is a third party application, so there is really not much we can do on that front anyway. Again, unfortunately upgrading is not an option for us at this point. Thanks, -Bill Caldarale, Charles R wrote: From: David Delbecq [mailto:[EMAIL PROTECTED] Subject: Re: Very Long Full GC after Inactivity check the jvm runs in server mode (check there is a '-server' parameter given to jvm startup). Server and client mode have different GC mecanisms. That's not correct; the set of GC algorithms available for server and client are identical, but the default heap sizes are different. For initial problem analysis, turn on -verbose:gc if you haven't already, and use JConsole to look at the heap usage. You may need a profiler to find out just what objects are hanging around and causing the long pause. You might want to try running with a later JVM - a 1.5 or 6 version should give you reduced pause times as well as better performance overall. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Very-Long-Full-GC-after-Inactivity-tf4589459.html#a13101470 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
From: Bill Clarke-Fields [mailto:[EMAIL PROTECTED] Subject: Re: Very Long Full GC after Inactivity Tomcat is running as a Windows service, and the jvm startup parameter is set to: C:\j2sdk1.4.2_11\jre\bin\server\jvm.dll. Do I also need to pass a -server parameter? No, there are no such parameters when running as a service. Your selection of the correct .dll guarantees server mode, which seems appropriate for your environment. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
From: Bill Clarke-Fields [mailto:[EMAIL PROTECTED] Subject: RE: Very Long Full GC after Inactivity The problem is that off-hours usage is very different, and leads to these very long full GCs. GC pause time is largely dependent on the number of live objects existing at the time of the GC, so yes, it could be an application problem. A profiler will help determine what these objects are. It's definitely unfortunate you can't upgrade, since that precludes you from running JConsole. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
I'm just throwing ideas out now, but would it help to set a lower value for the -Xms parameter? Would the JVM shrink the heap size down when usage is low? (I'm guessing not.) Or, is there a way to schedule (or force) full garbage collections? Thanks, -Bill Caldarale, Charles R wrote: From: Bill Clarke-Fields [mailto:[EMAIL PROTECTED] Subject: RE: Very Long Full GC after Inactivity The problem is that off-hours usage is very different, and leads to these very long full GCs. GC pause time is largely dependent on the number of live objects existing at the time of the GC, so yes, it could be an application problem. A profiler will help determine what these objects are. It's definitely unfortunate you can't upgrade, since that precludes you from running JConsole. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Very-Long-Full-GC-after-Inactivity-tf4589459.html#a13103292 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Very Long Full GC after Inactivity
From: Bill Clarke-Fields [mailto:[EMAIL PROTECTED] Subject: RE: Very Long Full GC after Inactivity I'm just throwing ideas out now, but would it help to set a lower value for the -Xms parameter? Probably not. For best throughput, you generally want -Xms and -Xmx to be the same, and as large as your environment allows. Would the JVM shrink the heap size down when usage is low? (I'm guessing not.) It does, but somewhat erratically. You really need to get a profiler running. Or, is there a way to schedule (or force) full garbage collections? Not guaranteed. You can call System.gc(), but there's no requirement that the JVM will honor the call (it usually does, though). If you install Lambda Probe (www.lambdaprobe.org), the System Information tab has a Force Garbage Collection link you can try. On the same page, Lambda Probe also has a Memory utilization button, but I don't remember if it works with your somewhat old versions of Tomcat and JVM. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Very Long Full GC after Inactivity
garbage collection is a reactive process, ie, it wont kick in until it really needs to. and the time it takes for the GC, depends on your java heap size, -Xmx, in JDK 1.5 you can specify GC goals like -XX:MaxGCMinorPauseMillis=value -XX:MaxGCPauseMillis=value to tell the JVM to try to align its GC to meet these goals, however, with JDK 1.4.2 you could try to make your heap smaller or to put in manual System.gc() at a periodic interval, neither of these options is what I like, I would just upgrade the JDK, even though you said its not an option, it would be your best bet for a list of all your jvm options at hand, please take a look at http://blogs.sun.com/watt/resource/jvm-options-list.html Filip Bill Clarke-Fields wrote: Hi, We are running an application on Tomcat 5.0.28 with Java 1.4.2. The usage of the application is very cyclical. It is used heavily during the day, and lightly at night. During peak daytime hours, a full garbage collection takes less than 2 seconds, which is fine. However, after a long period of inactivity in the evening, a full garbage collection will be triggered and take a very long time. Sometimes over a minute! Is there any way to avoid this long garbage collection? Why does it take so long? (Maybe because it is filling up with many small objects over time rather than a few larger objects?) Thanks for your help. -Bill p.s. Upgrading to a newer version of Tomcat or Java is unfortunately not an option at this point. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Very Long Full GC after Inactivity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bill, Bill Clarke-Fields wrote: Or, is there a way to schedule (or force) full garbage collections? Not really. You can issue a call to Runtime.gc(), but it's not guaranteed to do anything at all. My experience has been that the JVM does respect your request for a GC, even if it does take a few seconds to kick-in. Do you know what causes these 1-minute-long GCs to kick-off in the first place? I'm asking because you might be able to his a URL that would kick-start the full GC at, say, 5 in the morning just before your real customers got there. It' not perfect, but it might help. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHCqO59CaO5/Lv0PARAitPAJ9TkSr3a5s/ow70OkBVgodXtFyIFgCgi62l TsGjhWTkqGT/JNWPuFqUQoU= =9tx7 -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Very Long Full GC after Inactivity
Just a hypothesis. During these off-peak hours some other applications are actively running (maintenance etc.), and as java is not intensively used, its memory is swapped to the hard disks. Thus, when it awakens and does the GC, it takes so much time to wake up the memory walking through it to collect the garbage. What is memory consumption at night time? Are there memory-intensive tasks being performed at that time? May be restarting the tomcat can be scheduled after all those are done? (Or a forced GC, or filling memory with a bunch of dummy objects, up to Runtime.freeMemory()?) - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]