[SLUG] Memory profiler for web applications and other processes.
Hi all I have a virtual server under my control which runs a perl web application and now a Java Tomcat app which uses mail a fair bit. I'm loking for something which will tell me how much memory the perl app is taking versus the Tomcat app over a period of say 1 hour or so. Googling for 'memory profiler web applications' and things brings up things that you use to find memory leaks in apps which I dont want. Naturally top just gives me instantaneous values which don't mean much when a web app is only getting a few hits a minute or even less. Thats why I want to get an average over a few hours or so. Also I don't have Gnome or any gui thing on this server so it has to be command line or a perl or bash or other program that can be run from command line. Output to file would be perfect. Does anyone have suggestions? What do people here use for getting stats on programs like this? Mike -- Mike Lake Caver, Linux enthusiast and interested in anything technical. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
On 13/02/07, Mike Lake [EMAIL PROTECTED] wrote: Googling for 'memory profiler web applications' and things brings up things that you use to find memory leaks in apps which I dont want. Naturally top just gives me instantaneous values which don't mean much when a web app is only getting a few hits a minute or even less. Thats why I want to get an average over a few hours or so. Also I don't have Gnome or any gui thing on this server so it has to be command line or a perl or bash or other program that can be run from command line. Output to file would be perfect. Does anyone have suggestions? What do people here use for getting stats on programs like this? I'm not sure there is anything special about web applications - after all to the system they should look as just another process, although it usually generates lots of network traffic. exmap seems to be something about this, I haven't used it but from its Debian package dependencies it looks like it depends on GTK2 so it must be some sort of a GUI-based application. But maybe you can run it remotely with its window opened on your local $DISPLAY. --Amos -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
Dtrace? Just kidding. I'm interested in this answer too. We have similar probs. but between PHP/Apache and Apache Tomcat. Stu Director Polonious Pty Ltd (m) 0403 470 123 Polonious Support Numbers: Sydney: 61-2-9007-9842 Chicago: 1-312-212-3952 This above all: to thine ownself be true, And it must follow, as the night the day, Thou canst not then be false to any man. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
Amos Shapira wrote: On 13/02/07, Mike Lake [EMAIL PROTECTED] wrote: Googling for 'memory profiler web applications' and things brings up things that you use to find memory leaks in apps which I dont want. Naturally top just gives me instantaneous values which don't mean much when a web app is only getting a few hits a minute or even less. Thats why I want to get an average over a few hours or so. Also I don't have Gnome or any gui thing on this server so it has to be command line or a perl or bash or other program that can be run from command line. Output to file would be perfect. Does anyone have suggestions? What do people here use for getting stats on programs like this? I'm not sure there is anything special about web applications - after all to the system they should look as just another process, although it usually generates lots of network traffic. Yes, thats correct. I would just look at the sum total of all the Perl processes=the Perl app that is running vs sum of all Java stuff=the Tomcat. exmap seems to be something about this, I haven't used it but from its Debian package dependencies it looks like it depends on GTK2 so it must be some sort of a GUI-based application. But maybe you can run it remotely with its window opened on your local $DISPLAY. I found that using apt-cache on my laptop but to install it it will pull in GTK. As you mention it I have just looked it up on the web. Its using GTK for display only and its a perl script underneath that does the analysis. See its homepage at: http://www.berthels.co.uk/exmap/ Exmap is a tool to allow the real memory usage of a collection of processes to be examined. A linux kernel loadable module is used to export information to userspace, which is examined by a perl/gtk application to build a picture of how pages are shared amongst processes and their shared libraries. BUT! Exmap is linux-specific, since it uses a linux kernel loadable module. Additionally, the kernel module requires a fairly recent kernel (2.6.8 works, as may some earlier 2.6) in order to successfully compile or run. The server I have is a vserver running 2.4.22 kernel. So exmap is out anyways. Thanks for the suggestion. Mike -- Michael Lake Computational Research Support Unit Science Faculty, UTS Ph: 9514 2238 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
Michael, I would probably simply run a script something like 'while [ 1 ]; do date ps.log; ps axv | egrep perl|tomcat|apache ps.log;sleep 15; done' Then compare results over time (probably using a perl script to parse and accumulate the data you need) sar is also nice to log and give overall stats e.g. [EMAIL PROTECTED]:~$ sar -r 2 5 Linux 2.6.15-27-386 (reepy) 14/02/07 10:26:16kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 10:26:1848964142516 74.43 15372 46796 984856 19164 1.91 4 10:26:2048964142516 74.43 15372 46796 984856 19164 1.91 4 10:26:2248964142516 74.43 15376 46796 984856 19164 1.91 4 10:26:2448964142516 74.43 15376 46796 984856 19164 1.91 4 10:26:2648836142644 74.50 15376 46796 984856 19164 1.91 4 Average:48938142542 74.44 15374 46796 984856 19164 1.91 4 Regards, Martin On 2/14/07, Michael Lake [EMAIL PROTECTED] wrote: Amos Shapira wrote: On 13/02/07, Mike Lake [EMAIL PROTECTED] wrote: Googling for 'memory profiler web applications' and things brings up things that you use to find memory leaks in apps which I dont want. Naturally top just gives me instantaneous values which don't mean much when a web app is only getting a few hits a minute or even less. Thats why I want to get an average over a few hours or so. Also I don't have Gnome or any gui thing on this server so it has to be command line or a perl or bash or other program that can be run from command line. Output to file would be perfect. Does anyone have suggestions? What do people here use for getting stats on programs like this? I'm not sure there is anything special about web applications - after all to the system they should look as just another process, although it usually generates lots of network traffic. Yes, thats correct. I would just look at the sum total of all the Perl processes=the Perl app that is running vs sum of all Java stuff=the Tomcat. exmap seems to be something about this, I haven't used it but from its Debian package dependencies it looks like it depends on GTK2 so it must be some sort of a GUI-based application. But maybe you can run it remotely with its window opened on your local $DISPLAY. I found that using apt-cache on my laptop but to install it it will pull in GTK. As you mention it I have just looked it up on the web. Its using GTK for display only and its a perl script underneath that does the analysis. See its homepage at: http://www.berthels.co.uk/exmap/ Exmap is a tool to allow the real memory usage of a collection of processes to be examined. A linux kernel loadable module is used to export information to userspace, which is examined by a perl/gtk application to build a picture of how pages are shared amongst processes and their shared libraries. BUT! Exmap is linux-specific, since it uses a linux kernel loadable module. Additionally, the kernel module requires a fairly recent kernel (2.6.8 works, as may some earlier 2.6) in order to successfully compile or run. The server I have is a vserver running 2.4.22 kernel. So exmap is out anyways. Thanks for the suggestion. Mike -- Michael Lake Computational Research Support Unit Science Faculty, UTS Ph: 9514 2238 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- Regards, Martin Martin Visser -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
quote who=Stuart Guthrie Dtrace? Just kidding. Why kidding? It's an exceptional tool for the job. How much does your Open Source operating system choice *really* matter [1]? It's worth changing the complement to your work if it's not helping you! - Jeff [1] Okay, so 'management' is a really good answer to this. You could always run multiple testing platforms. -- Open CeBIT 2007: Sydney, Australia http://www.opencebit.com.au/ GDK (acronym): GNU's Not Unix Image Manipulation Program Tool-Kit Drawing-Kit. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
On Wednesday 14 February 2007 08:05, [EMAIL PROTECTED] wrote: I have a virtual server under my control which runs a perl web application and now a Java Tomcat app which uses mail a fair bit. I'm loking for something which will tell me how much memory the perl app is taking versus the Tomcat app over a period of say 1 hour or so. Googling for 'memory profiler web applications' and things brings up things that you use to find memory leaks in apps which I dont want. Naturally top just gives me instantaneous values which don't mean much when a web app is only getting a few hits a minute or even less. Thats why I want to get an average over a few hours or so. Also I don't have Gnome or any gui thing on this server so it has to be command line or a perl or bash or other program that can be run from command line. Output to file would be perfect. Does anyone have suggestions? What do people here use for getting stats on programs like this? If you are finding it hard ... it's because there is no such thing! An app running consists of [the app]-usually small [shared libraries]-usually big. So 10 apps is a bit more than 1 app. (10*app + shared libraries) At any one instant memory-usage-on-a-system vs system+app1 and system+app2 is a broad fuzzy kind of general hint. EG an app may use vast amounts of mem that is never touched and is safely ignored in swap. So it would look like a memory pig but really be a very attractive (efficiency wise) program. EG kde is much more memory hungry than icewm. How much? well lots! Does it show? Depends on what else the system is doing! The metric you are seeking could easily be totally misleading. If you insist on doing this study the output of top until you are bored. UNIX is not like windows. A metric on one does not compare to the other. Best advice: lower the physical ram until your system swaps. compare apps. James -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
You might try dstat - http://dag.wieers.com/home-made/dstat/ -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Memory profiler for web applications and other processes.
On Wed, 2007-02-14 at 12:04 +1100, Matthew Hannigan wrote: You might try dstat - http://dag.wieers.com/home-made/dstat/ Cute! Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. signature.asc Description: This is a digitally signed message part -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
