Re: performance testing - emulating real world use
My experience with commercial load-testing apps is that they are outrageously expensive, a pain to program, don't really scale all that well, and mostly have to run on Windows with someone sitting at the mouse. There are some that work better than others, but the free stuff in this areas is quite good. Ditto. I've found it easier to hack together something with LWP's LWP::UserAgent and Benchmark. Particularly when load-testing an application that required different pre-registered users that had not performed this particular transaction we were load testing. (A survey with random and different questions - depending on the user!). None of the macro-like testing apps could do very much with regard to that kind of interaction and variability in the content generated by the application we were load-testing. But a 30-line Perl script that simply appended the Benchmark results into a tab-delimited file worked great. We found about 30 instances of Perl running the script per WinPC ate the machine. So we only loaded 15 per PC during actual testing and added more distributed nodes to the test. As an aside, the whole thing was an exercise in needing a cup of sugar and asking the local grocery store how much sugar they have on the shelves. i.e. What is the point of measuring beyond the more than the 1 cup you need? So we measured (at great expense) and determined that the entire lifetime load (~1yr) expected for all users on their system could be accomplished on the existing sytem during a lunch hour. Mike808/ - http://www.valuenet.net
Re: performance testing - emulating real world use
Jauder Ho wrote: Another application (commercial) is Mercury Interactive's LoadRunner. My experience with commercial load-testing apps is that they are outrageously expensive, a pain to program, don't really scale all that well, and mostly have to run on Windows with someone sitting at the mouse. There are some that work better than others, but the free stuff in this areas is quite good. I recommend httperf and http_load for banging on lists of URLs really hard. At eToys, one of our developers rigged up some shell scripts that would play back log files through httperf and that worked pretty well. If you want to record browser sessions for testing specific paths through the site, look at http://sourceforge.net/projects/http-recorder/ or http://sourceforge.net/projects/roboweb/. There's also webchatpp, HTTP::WebTest, and HTTP::MonkeyWrench on CPAN. All of these have been discussed on this list before. - Perrin
performance testing - emulating real world use
Anyone know of good guides or general info on performance testing and emulating real use of an application. I would like to understand how to identify potential bottlenecks before I deploy web apps. thank you, ~ b r y a n
Re: performance testing - emulating real world use
Bryan Henry wrote: Anyone know of good guides or general info on performance testing and emulating real use of an application. I would like to understand how to identify potential bottlenecks before I deploy web apps. thank you, ~ b r y a n try httpd.apache.org/test/ and perl framework there in as well look on freshmeat for siege it does testing too
Re: performance testing - emulating real world use
On Tue, Mar 12, 2002 at 01:52:36PM -0800, clayton cottingham wrote: Bryan Henry wrote: Anyone know of good guides or general info on performance testing and emulating real use of an application. I would like to understand how to identify potential bottlenecks before I deploy web apps. thank you, ~ b r y a n I've used HTTPD::Bench::ApacheBench (available on CPAN) to do load testing. It seems to do a good job. The hardest part is writing the testing script (especially for form transactions..). However, if you can do your requests with LWP it's fairly straightforward to convert over to the ApacheBench data structures. I'm considering writing a little mod_perl proxy server that records the different transactions. Then I could just munge the separate Authorization: headers to do some serious load testing... try httpd.apache.org/test/ and perl framework there in as well look on freshmeat for siege it does testing too -- Paul Lindner[EMAIL PROTECTED] | | | | | | | | | | mod_perl Developer's Cookbook http://www.modperlcookbook.org/ Human Rights Declaration http://www.unhchr.ch/udhr/
Re: performance testing - emulating real world use
Heyas, BHAnyone know of good guides or general info on BHperformance testing and emulating real use of BHan application. As a general rule, it's easiest if you have a production system already running. Record all information that you need to reproduce the requests (typically, HTTP request headers and POST data if applicable), from a production server and you can replay any amount of data on a sandboxed QA environment. You can either eliminate or proportionally shorten the time period between requests to space out load arbitrarily. This is extremely effective if you have enough real user data because you're not inventing user load. You're using real user load. I don't know of any product that does this all at once, but it's not hard to hack together. If your site is entirely GET based, you can probably just make do with parsing access logs and turning those into requests. I believe Apache::DumpHeaders might get you most of the way on the capturing side if you need special headers, cookies, or POST information. Feeding scripts into commercial products like SilkPerformer will give you the best client side testing and reports. However, a homebrew Perl user agent will do okay, too. Unfortunately, ab doesn't support taking in a custom URL workload. For a simple record/replay load test tool that works remarkably well, check out the resource CD that ships with Windows 2000 and you will find the Microsoft Web Stress Tester. It's free and GUI based and can record IE sessions and replay them with an arbitrary number of threads. It uses Access databases to hold the tests and results so you can probably use Perl on Windows to populate it with your custom tests. Humbly, Andrew -- Andrew Ho http://www.tellme.com/ [EMAIL PROTECTED] Engineer [EMAIL PROTECTED] Voice 650-930-9062 Tellme Networks, Inc. 1-800-555-TELLFax 650-930-9101 --
Re: performance testing - emulating real world use
On Tue, 12 Mar 2002, Andrew Ho wrote: [...] This is extremely effective if you have enough real user data because you're not inventing user load. You're using real user load. Not really; you also have to emulate the connection speeds of the users. Or does the tools you mentioned do that? - ask -- ask bjoern hansen, http://ask.netcetera.dk/ !try; do(); more than a billion impressions per week, http://valueclick.com
Re: performance testing - emulating real world use
Hello, ABHNot really; you also have to emulate the connection speeds of the ABHusers. Or does the tools you mentioned do that? Both of the commercially produced tools I mentioned (SilkPerformer and the free Microsoft Web Stress program) can throttle bandwidth. Rolling your own is a bunch harder. So you're correct. My point though is not so much that the load profile of what pages get loaded in what order, and what data calls and dynamic scripts are run in what order are genuine. If you simulate the timing between requests, you'll even get spikes that are similar to the real thing. It's definitely not reality! You also miss anomalies like users closing browsers and (unless you capture full headers) which clients support keep-alives for example. But, it's closer to reality than most scripts that are invented (especially be developers ;)). Humbly, Andrew -- Andrew Ho http://www.tellme.com/ [EMAIL PROTECTED] Engineer [EMAIL PROTECTED] Voice 650-930-9062 Tellme Networks, Inc. 1-800-555-TELLFax 650-930-9101 --
Re: performance testing - emulating real world use
Hello, AHSo you're correct. My point though is not so much that the load profile of AHwhat pages get loaded in what order, and what data calls and dynamic AHscripts are run in what order are genuine. If you simulate the timing AHbetween requests, you'll even get spikes that are similar to the real AHthing. It's definitely not reality! You also miss anomalies like users AHclosing browsers and (unless you capture full headers) which clients AHsupport keep-alives for example. But, it's closer to reality than most AHscripts that are invented (especially be developers ;)). Man, I can't type worth anything today. The gist of what I meant to type was this: the exact load of the production server will not be replicated in your simulation; but the load from data calls and dynamically generated content will be similar in nature, patterned after how your server is hit in real life. This will likely be a better exercise of your server than a developer-invented test script. If you have a production environment (or proxy) set up that can capture real user requests, this is also far less work for creating a convincing simulation load than having to sit down and write a new script every time your application changes. Humbly, Andrew -- Andrew Ho http://www.tellme.com/ [EMAIL PROTECTED] Engineer [EMAIL PROTECTED] Voice 650-930-9062 Tellme Networks, Inc. 1-800-555-TELLFax 650-930-9101 --
Re: performance testing - emulating real world use
Another application (commercial) is Mercury Interactive's LoadRunner. It actually records events and plays it back on load generator machines. It's fairly complex, has LOTs of knobs to turn and can load test quite a bit more than just web apps, I use it to load test/benchmark Oracle 11i for instance. The software is not cheap but definitely worth looking into if you are serious about testing. (www.merc-int.com) They also sell something called ActiveTest which may be more suited to a web applications. In this case, they will test your site for you using their hardware at a colo site. --Jauder On Tue, 12 Mar 2002, Andrew Ho wrote: Heyas, BHAnyone know of good guides or general info on BHperformance testing and emulating real use of BHan application. As a general rule, it's easiest if you have a production system already running. Record all information that you need to reproduce the requests (typically, HTTP request headers and POST data if applicable), from a production server and you can replay any amount of data on a sandboxed QA environment. You can either eliminate or proportionally shorten the time period between requests to space out load arbitrarily. This is extremely effective if you have enough real user data because you're not inventing user load. You're using real user load. I don't know of any product that does this all at once, but it's not hard to hack together. If your site is entirely GET based, you can probably just make do with parsing access logs and turning those into requests. I believe Apache::DumpHeaders might get you most of the way on the capturing side if you need special headers, cookies, or POST information. Feeding scripts into commercial products like SilkPerformer will give you the best client side testing and reports. However, a homebrew Perl user agent will do okay, too. Unfortunately, ab doesn't support taking in a custom URL workload. For a simple record/replay load test tool that works remarkably well, check out the resource CD that ships with Windows 2000 and you will find the Microsoft Web Stress Tester. It's free and GUI based and can record IE sessions and replay them with an arbitrary number of threads. It uses Access databases to hold the tests and results so you can probably use Perl on Windows to populate it with your custom tests. Humbly, Andrew -- Andrew Ho http://www.tellme.com/ [EMAIL PROTECTED] Engineer [EMAIL PROTECTED] Voice 650-930-9062 Tellme Networks, Inc. 1-800-555-TELLFax 650-930-9101 --
Re: performance testing - emulating real world use
On Tue, 12 Mar 2002, Jauder Ho wrote: Another application (commercial) is Mercury Interactive's LoadRunner. It actually records events and plays it back on load generator machines. It's fairly complex, has LOTs of knobs to turn and can load test quite a bit more than just web apps, I use it to load test/benchmark Oracle 11i for instance. The software is not cheap but definitely worth looking into if you are serious about testing. (www.merc-int.com) They also sell something called ActiveTest which may be more suited to a web applications. In this case, they will test your site for you using their hardware at a colo site. Before anyone even looks into this, be warned they quoted me £50,000 once for LoadRunner. Needless to say I was flabbergasted (though their software did look kinda cool). -- !-- Matt -- :-Get a smart net/:-
Re: performance testing - emulating real world use
Heh. Forgot to state that it does cost an arm and a leg but it's one of the few software packages that is worth considering paying money for IMO. However, with the economy being the way it is, it is possible to rent the software for a period of time but this is done by special arrangement on a case by case basis. If you ask, they may be willing to give you a copy of the software. The out-the-box install allows you to record and playback as well as load up to 10 (iirc) users which nicely lets you test out the functionality. I think I have a copy of it somewhere around here. If there is interest, I can get you in touch with the right people. --Jauder On Wed, 13 Mar 2002, Matt Sergeant wrote: On Tue, 12 Mar 2002, Jauder Ho wrote: Another application (commercial) is Mercury Interactive's LoadRunner. It actually records events and plays it back on load generator machines. It's fairly complex, has LOTs of knobs to turn and can load test quite a bit more than just web apps, I use it to load test/benchmark Oracle 11i for instance. The software is not cheap but definitely worth looking into if you are serious about testing. (www.merc-int.com) They also sell something called ActiveTest which may be more suited to a web applications. In this case, they will test your site for you using their hardware at a colo site. Before anyone even looks into this, be warned they quoted me £50,000 once for LoadRunner. Needless to say I was flabbergasted (though their software did look kinda cool). -- !-- Matt -- :-Get a smart net/:-
