Re: Huge tomcat memory footprint
Anand The pmap command itself gives some good info on this huge memory footrpint. There will be jars, some libraries (JVM specifc) are loaded into the memory that can cause this footprint. You can find the list of jars/files loaded by using the inode number. In addition sometime i observed that if the System memory (RAM) is low, it tends to show a larger footprint. -Surendra - Original Message - From: "Anand Narasimhan" <[EMAIL PROTECTED]> To: "'Tomcat Users List'" <[EMAIL PROTECTED]> Sent: Thursday, November 04, 2004 11:52 PM Subject: RE: Huge tomcat memory footprint > > > > -Original Message- > > From: Shapira, Yoav [mailto:[EMAIL PROTECTED] > > Sent: Thursday, November 04, 2004 9:59 AM > > To: Tomcat Users List > > Subject: RE: Huge tomcat memory footprint > > > > > > Hi, > > BTW, have you moved from JDK 1.4.1 to 1.4.2 yet, and if so > > has that made a difference? IIRC your OP said you're using > > JDK 1.4.1.. > > > > Yoav Shapira http://www.yoavshapira.com > > > > > > Yes. I did upgrade to JDK 1.4.2_06. It did not make much difference. In fact > it increased the memory footprint. > > Anand > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
> -Original Message- > From: Shapira, Yoav [mailto:[EMAIL PROTECTED] > Sent: Thursday, November 04, 2004 9:59 AM > To: Tomcat Users List > Subject: RE: Huge tomcat memory footprint > > > Hi, > BTW, have you moved from JDK 1.4.1 to 1.4.2 yet, and if so > has that made a difference? IIRC your OP said you're using > JDK 1.4.1.. > > Yoav Shapira http://www.yoavshapira.com > > Yes. I did upgrade to JDK 1.4.2_06. It did not make much difference. In fact it increased the memory footprint. Anand - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
Hi, BTW, have you moved from JDK 1.4.1 to 1.4.2 yet, and if so has that made a difference? IIRC your OP said you're using JDK 1.4.1.. Yoav Shapira http://www.yoavshapira.com >-Original Message- >From: Anand Narasimhan [mailto:[EMAIL PROTECTED] >Sent: Thursday, November 04, 2004 12:54 PM >To: 'Tomcat Users List' >Subject: RE: Huge tomcat memory footprint > >Hi, > >> >80MB >> >and heap used was 57MB (reported by optimizeit). But the >> memory sizes >> >reported by solairs pmap command is >> > >> >Size RSSShared Private >> >1173.45 MB608.125 MB 15.125 MB 593 MB >> >> There's a mismatch here. There's no way the OS-level >> overhead is hundreds of MBs for an 80-MB heap. Are you sure >> the pmap command doesn't multiply by the number of threads or >> something like that (similar to what the Linux top command does)? >> > >I am not sure how pmap works. But the numbers are consitent with what top >command reports. > >> >> >3. I read somewhere that if you use reflection, the reflective data >> >structures (whatever that means) also contribute to the memory >> footprint. >> >It >> >sounded as though these are not part of the heap. So if we >> are creating >> >instances of Method, Constructor, etc, are they not allocated in the >> heap? >> >We are using reflection heavily. >> >> Where's "somewhere" ? ;) That's mostly old information. The >> JVM is very efficient at handling reflection and properly >> storing Method references. > >I think I read about this in the Java performance tuning book from Sun >press >or O'reilly. > > >Anand > > >- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
Hi, > >80MB > >and heap used was 57MB (reported by optimizeit). But the > memory sizes > >reported by solairs pmap command is > > > >Size RSSShared Private > >1173.45 MB608.125 MB 15.125 MB 593 MB > > There's a mismatch here. There's no way the OS-level > overhead is hundreds of MBs for an 80-MB heap. Are you sure > the pmap command doesn't multiply by the number of threads or > something like that (similar to what the Linux top command does)? > I am not sure how pmap works. But the numbers are consitent with what top command reports. > > >3. I read somewhere that if you use reflection, the reflective data > >structures (whatever that means) also contribute to the memory > footprint. > >It > >sounded as though these are not part of the heap. So if we > are creating > >instances of Method, Constructor, etc, are they not allocated in the > heap? > >We are using reflection heavily. > > Where's "somewhere" ? ;) That's mostly old information. The > JVM is very efficient at handling reflection and properly > storing Method references. I think I read about this in the Java performance tuning book from Sun press or O'reilly. Anand - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
Hi, >80MB >and heap used was 57MB (reported by optimizeit). But the memory sizes >reported by solairs pmap command is > >Size RSSShared Private >1173.45 MB608.125 MB 15.125 MB 593 MB There's a mismatch here. There's no way the OS-level overhead is hundreds of MBs for an 80-MB heap. Are you sure the pmap command doesn't multiply by the number of threads or something like that (similar to what the Linux top command does)? > >of few hours. Ofcourse, optimizeit itself has overheads and adds to the A profiler typically adds an order of magnitude overhead. So for an 80MB heap on the app normally, an 800MB heap with a profiler is within the bounds of reason. But that'd be within the heap and the JVM. >process on solaris is 4GB. What will happen when the JVM process size >reaches 4GB? Changes are it'll crash. >2. I see that GC is running, reclaiming memory periodically. But I noticed >that after a while GC is running more frequently and each run takes a >substantial amount of time (in one instance GC took about 14 secs). This >may >affect the throughput, but will it have any impact on the memory? If GC can't free up memory, then not only will your throughput suffer but it'll be for naught. This is a classic memory leak scenario. >3. I read somewhere that if you use reflection, the reflective data >structures (whatever that means) also contribute to the memory footprint. >It >sounded as though these are not part of the heap. So if we are creating >instances of Method, Constructor, etc, are they not allocated in the heap? >We are using reflection heavily. Where's "somewhere" ? ;) That's mostly old information. The JVM is very efficient at handling reflection and properly storing Method references. As you said, you need to look at the profiler more carefully. Yoav This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
Hi, Thanks again for your suggestions. I still don't have a good handle on the problem. So please bear with me If I am asking the same questions again. 1. What I have noticed so far is the heap is not growing too much, but the process foot print is growing. I used optimizeit memory profiler to compare snapshots of the heap. I don't see any obvious memory leaks. I need to take a closer look. When I checked this morning, the heap size allocated was 80MB and heap used was 57MB (reported by optimizeit). But the memory sizes reported by solairs pmap command is Size RSSShared Private 1173.45 MB608.125 MB 15.125 MB 593 MB I understand that the process footprint includes other data and would grow as JVM itself grows. But is it normal that it grows to 1173 MB in a matter of few hours. Ofcourse, optimizeit itself has overheads and adds to the process size. But I have seen it grow to 2.5 GB in two days without optimizeit when I was doing some stress tests. I read in one of the solaris virtual memory architecture design paper, that virtual address space of a process on solaris is 4GB. What will happen when the JVM process size reaches 4GB? I guess my confusion is due to the lack of my understanding on how java manages memory and how it relates to the OS-level memory management. Are there any documents or white papers that discuss this kind of issues? 2. I see that GC is running, reclaiming memory periodically. But I noticed that after a while GC is running more frequently and each run takes a substantial amount of time (in one instance GC took about 14 secs). This may affect the throughput, but will it have any impact on the memory? 3. I read somewhere that if you use reflection, the reflective data structures (whatever that means) also contribute to the memory footprint. It sounded as though these are not part of the heap. So if we are creating instances of Method, Constructor, etc, are they not allocated in the heap? We are using reflection heavily. Thanks Anand > -Original Message- > From: Shapira, Yoav [mailto:[EMAIL PROTECTED] > Sent: Thursday, November 04, 2004 4:55 AM > To: Tomcat Users List > Subject: RE: Huge tomcat memory footprint > > > Hi, > > >If there is a memory leak in our application, tomcat or > java, what are > the > >symptoms I should look for. > >Will the heap usage (shown by visual GC, optimizeit etc) keep growing > and > >eventually reach the max limit set using -Xmx option to java? > > Yes, the heap will keep growing and eventually reach the max, > at which point you will get OutOfMemoryErrors and > hung/unstable behavior. > However, if your leak is small, it might take a very long > time before this situation is reached. > > Run your app inside a profiler, conduct some operations > typical to your app, and then look at the heap. Most > profilers now have features that show you object reference > trees, i.e. what objects are holding onto what references. > If you see stuff still in memory that you think shouldn't be > there, fix it. > > Yoav > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
Hi, >If there is a memory leak in our application, tomcat or java, what are the >symptoms I should look for. >Will the heap usage (shown by visual GC, optimizeit etc) keep growing and >eventually reach the max limit set using -Xmx option to java? Yes, the heap will keep growing and eventually reach the max, at which point you will get OutOfMemoryErrors and hung/unstable behavior. However, if your leak is small, it might take a very long time before this situation is reached. Run your app inside a profiler, conduct some operations typical to your app, and then look at the heap. Most profilers now have features that show you object reference trees, i.e. what objects are holding onto what references. If you see stuff still in memory that you think shouldn't be there, fix it. Yoav This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Huge tomcat memory footprint
Hi Anand, We currently have a similar problem. Under load the JVM consumes more and more memory. (We are using tomcat 5.0 and Sun JVM 1.4.2 and linux (debian sarge)) In our case it looks as if the Garbage Collector is not getting enough time to free memory - but this is still very much a guess. You may want to have a look in this direction however. We are currently looking at see what setting we can tune for the GC or maybe using a different JVM. Good luck, Andrew > > >-Original Message- > > >From: Anand Narasimhan [mailto:[EMAIL PROTECTED] > > >Sent: Monday, November 01, 2004 5:26 PM > > >To: [EMAIL PROTECTED] > > >Subject: Huge tomcat memory footprint > > > > > >Hi, > > > > > >I am not able to determine if the problem is related to > > > > tomcat, solaris > > or > > > > >the application itself. Any help/pointers to debug the > > > > problem will be > > > > >greatly appreciated. > > > > > >The application runs on Solaris 2.8 using Sun's Java version > > > > 1.4.1_02. > > The > > > > >tomcat version is 4.1.27. The GUI is written using HTML/JSP/Struts > > > > 1.0.2. > > > > >The database is Oracle version 8.x (I think). The > > > > application also has > > API > > > > >interface written using Java/XML/Soap etc. The application is > > > > memory/CPU > > > > >intensive. The problem I am having is, after running under a large > > > > load, > > > > >tomcat process's memory footprint (reported by top command) > > > > increases > > > > >rapidly to more that 2G. The heap size (max configured to 1G), > > > > increases to > > > > >about 500 - 600M. I have tried running tomcat with > > > > optimizeit to see if > > > > >there are any memory leaks. optimizeit as wells as the > > > > output from GC > > > > >(running with -verbose:gc, -XX+PrintGCDetails) shows > > > > frequent garbage > > > > >collection activity and the heap size does not grow too much. > > > > > >I am not able to figure out why the process memory grows. If the heap > > > > is > > > > >not > > >growing too much, what is consuming the memory? > > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
If there is a memory leak in our application, tomcat or java, what are the symptoms I should look for. Will the heap usage (shown by visual GC, optimizeit etc) keep growing and eventually reach the max limit set using -Xmx option to java? Thanks Anand > -Original Message- > From: Shapira, Yoav [mailto:[EMAIL PROTECTED] > Sent: Tuesday, November 02, 2004 5:49 AM > To: Tomcat Users List > Subject: RE: Huge tomcat memory footprint > > > Hi, > Yup, upgrade the JDK if you can, that's a bad version. > Struts 1.0.2 is also fairly ancient, you'll gain if you > update that as well. > > The heap is not the only memory consumer in a given JVM. > There's the stack, symbol tables, various lookup tables, etc. > As you load classes at runtime some of these areas grow in > consumption. That's normal and expected. > > Then there's the OS-level memory, which is different and > always larger than the JVM itself. That's why top is a > sub-optimal (unless you really know what you're doing) > indicator of JVM memory usage. The OS-level memory includes > other data such as paging and thread management information, > in addition to anything and everything used by the JVM. As > the JVM itself grows, so will the OS-level memory consumption. > > Yoav Shapira http://www.yoavshapira.com > > > >-Original Message- > >From: Anand Narasimhan [mailto:[EMAIL PROTECTED] > >Sent: Monday, November 01, 2004 5:26 PM > >To: [EMAIL PROTECTED] > >Subject: Huge tomcat memory footprint > > > >Hi, > > > >I am not able to determine if the problem is related to > tomcat, solaris > or > >the application itself. Any help/pointers to debug the > problem will be > >greatly appreciated. > > > >The application runs on Solaris 2.8 using Sun's Java version > 1.4.1_02. > The > >tomcat version is 4.1.27. The GUI is written using HTML/JSP/Struts > 1.0.2. > >The database is Oracle version 8.x (I think). The > application also has > API > >interface written using Java/XML/Soap etc. The application is > memory/CPU > >intensive. The problem I am having is, after running under a large > load, > >tomcat process's memory footprint (reported by top command) > increases > >rapidly to more that 2G. The heap size (max configured to 1G), > increases to > >about 500 - 600M. I have tried running tomcat with > optimizeit to see if > >there are any memory leaks. optimizeit as wells as the > output from GC > >(running with -verbose:gc, -XX+PrintGCDetails) shows > frequent garbage > >collection activity and the heap size does not grow too much. > > > >I am not able to figure out why the process memory grows. If the heap > is > >not > >growing too much, what is consuming the memory? > > > >Thanks > >Anand > > > >_ > > > >Anand Narasimhan > >[EMAIL PROTECTED] > > > > > > This e-mail, including any attachments, is a confidential > business communication, and may contain information that is > confidential, proprietary and/or privileged. This e-mail is > intended only for the individual(s) to whom it is addressed, > and may not be saved, copied, printed, disclosed or used by > anyone else. If you are not the(an) intended recipient, > please immediately delete this e-mail from your computer > system and notify the sender. Thank you. > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
Thanks for the reply. > -Original Message- > From: Shapira, Yoav [mailto:[EMAIL PROTECTED] > Sent: Tuesday, November 02, 2004 5:49 AM > To: Tomcat Users List > Subject: RE: Huge tomcat memory footprint > > > Hi, > Yup, upgrade the JDK if you can, that's a bad version. > Struts 1.0.2 is also fairly ancient, you'll gain if you > update that as well. I have upgraded the JDK version to 1.4.2_06 and ran some small test cases. But I don't see a significant change. > > The heap is not the only memory consumer in a given JVM. > There's the stack, symbol tables, various lookup tables, etc. > As you load classes at runtime some of these areas grow in > consumption. That's normal and expected. > > Then there's the OS-level memory, which is different and > always larger than the JVM itself. That's why top is a > sub-optimal (unless you really know what you're doing) > indicator of JVM memory usage. The OS-level memory includes > other data such as paging and thread management information, > in addition to anything and everything used by the JVM. As > the JVM itself grows, so will the OS-level memory consumption. I understand that the process size will be a lot larger than the JVM itself. But in this case the size keeps growing. At one stage I saw that the size reported by top, pmap and pmem commands was about 2.5G. Is this normal. I have 1G RAM on my system. If the process virtual memory keeps growing, doesn't the OS needs to do frequent paging and hence affect the performance of the system? > > Yoav Shapira http://www.yoavshapira.com > > > >-Original Message- > >From: Anand Narasimhan [mailto:[EMAIL PROTECTED] > >Sent: Monday, November 01, 2004 5:26 PM > >To: [EMAIL PROTECTED] > >Subject: Huge tomcat memory footprint > > > >Hi, > > > >I am not able to determine if the problem is related to > tomcat, solaris > or > >the application itself. Any help/pointers to debug the > problem will be > >greatly appreciated. > > > >The application runs on Solaris 2.8 using Sun's Java version > 1.4.1_02. > The > >tomcat version is 4.1.27. The GUI is written using HTML/JSP/Struts > 1.0.2. > >The database is Oracle version 8.x (I think). The > application also has > API > >interface written using Java/XML/Soap etc. The application is > memory/CPU > >intensive. The problem I am having is, after running under a large > load, > >tomcat process's memory footprint (reported by top command) > increases > >rapidly to more that 2G. The heap size (max configured to 1G), > increases to > >about 500 - 600M. I have tried running tomcat with > optimizeit to see if > >there are any memory leaks. optimizeit as wells as the > output from GC > >(running with -verbose:gc, -XX+PrintGCDetails) shows > frequent garbage > >collection activity and the heap size does not grow too much. > > > >I am not able to figure out why the process memory grows. If the heap > is > >not > >growing too much, what is consuming the memory? > > > >Thanks > >Anand > > > >_ > > > >Anand Narasimhan > >[EMAIL PROTECTED] > > > > > > This e-mail, including any attachments, is a confidential > business communication, and may contain information that is > confidential, proprietary and/or privileged. This e-mail is > intended only for the individual(s) to whom it is addressed, > and may not be saved, copied, printed, disclosed or used by > anyone else. If you are not the(an) intended recipient, > please immediately delete this e-mail from your computer > system and notify the sender. Thank you. > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Huge tomcat memory footprint
Hi, Yup, upgrade the JDK if you can, that's a bad version. Struts 1.0.2 is also fairly ancient, you'll gain if you update that as well. The heap is not the only memory consumer in a given JVM. There's the stack, symbol tables, various lookup tables, etc. As you load classes at runtime some of these areas grow in consumption. That's normal and expected. Then there's the OS-level memory, which is different and always larger than the JVM itself. That's why top is a sub-optimal (unless you really know what you're doing) indicator of JVM memory usage. The OS-level memory includes other data such as paging and thread management information, in addition to anything and everything used by the JVM. As the JVM itself grows, so will the OS-level memory consumption. Yoav Shapira http://www.yoavshapira.com >-Original Message- >From: Anand Narasimhan [mailto:[EMAIL PROTECTED] >Sent: Monday, November 01, 2004 5:26 PM >To: [EMAIL PROTECTED] >Subject: Huge tomcat memory footprint > >Hi, > >I am not able to determine if the problem is related to tomcat, solaris or >the application itself. Any help/pointers to debug the problem will be >greatly appreciated. > >The application runs on Solaris 2.8 using Sun's Java version 1.4.1_02. The >tomcat version is 4.1.27. The GUI is written using HTML/JSP/Struts 1.0.2. >The database is Oracle version 8.x (I think). The application also has API >interface written using Java/XML/Soap etc. The application is memory/CPU >intensive. The problem I am having is, after running under a large load, >tomcat process's memory footprint (reported by top command) increases >rapidly to more that 2G. The heap size (max configured to 1G), increases to >about 500 - 600M. I have tried running tomcat with optimizeit to see if >there are any memory leaks. optimizeit as wells as the output from GC >(running with -verbose:gc, -XX+PrintGCDetails) shows frequent garbage >collection activity and the heap size does not grow too much. > >I am not able to figure out why the process memory grows. If the heap is >not >growing too much, what is consuming the memory? > >Thanks >Anand > >_ > >Anand Narasimhan >[EMAIL PROTECTED] > This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Huge tomcat memory footprint
Hi, your problem is Java 1.4.1_02, it has a memory leak in StringBuffer which is used a lot for XML parsing. Upgrade to 1.4.2 and it will be fine. Pete On Monday 01 November 2004 22:25, Anand Narasimhan wrote: > Hi, > > I am not able to determine if the problem is related to tomcat, solaris or > the application itself. Any help/pointers to debug the problem will be > greatly appreciated. > > The application runs on Solaris 2.8 using Sun's Java version 1.4.1_02. The > tomcat version is 4.1.27. The GUI is written using HTML/JSP/Struts 1.0.2. > The database is Oracle version 8.x (I think). The application also has API > interface written using Java/XML/Soap etc. The application is memory/CPU > intensive. The problem I am having is, after running under a large load, > tomcat process's memory footprint (reported by top command) increases > rapidly to more that 2G. The heap size (max configured to 1G), increases to > about 500 - 600M. I have tried running tomcat with optimizeit to see if > there are any memory leaks. optimizeit as wells as the output from GC > (running with -verbose:gc, -XX+PrintGCDetails) shows frequent garbage > collection activity and the heap size does not grow too much. > > I am not able to figure out why the process memory grows. If the heap is > not growing too much, what is consuming the memory? > > Thanks > Anand > > _ > > Anand Narasimhan > [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
