RE: load testing (was RE: Difficult To Reproduce Session Issue)
> I would (did) take WAST over any of those tools (except > OpenSTA because I've never used it). I don't know about > OpenSTA, but LoadRunner, SilkPerformer and Empirix are all > pretty beastly in comparison. I actually found SilkPerformer to be pretty easy to use, for what that's worth. I also found the scripting capabilities it provided to be both easy and extremely flexible, if you remember any Pascal. It's not cheap, though; far from it. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ voice: (202) 797-5496 fax: (202) 797-5444 __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
I would (did) take WAST over any of those tools (except OpenSTA because I've never used it). I don't know about OpenSTA, but LoadRunner, SilkPerformer and Empirix are all pretty beastly in comparison. Empirix requires too many machines to simulate a heavy client load. They are huge, expensive, and complicated. There are certainly some interesting things they can do (mostly nice-to-have features, like fancy reports or CORBA and EJB client support) that WAST can't, but a complete COM API, full control of headers, ports, remote clients, item grouping and weighting, storage of results in an Access d.b. with sensible table names, and the price make WAST unbeatable for flexibility and unrestricted deployment (except all your clients have to be Win32). Like I said, there's also Apache JMeter, which is also free and seems to be getting a lot more use judging from their mailing list. I don't know anything about the mailing list's properties. It's maintained by Mike Dinowotz at http://www.houseoffusion.com Scott > -Original Message- > From: charles arehart [mailto:[EMAIL PROTECTED]] > > You raise good points, Scott. Certainly, to test a site > inside a firewall, > you'll need to install some software. But if that's not a > concern, and if > you want to avoid installing (and learning) some software, this hosted > solution is still a worthwhile alternative. It's dead easy. > And even for > just 5 users, it could be useful as Chris noted. > > I previously installed by WAST and OpenSTA (as well as > Empirix E-test Suite, > Mercury Interactive's Astra QuickTest, and Parasoft WebKing). > And I found > they all had complications that kept me from using them on an > ongoing basis. > > I'm just raising this as an alternative for any who may have > been through > the same process. > > /charlie > > PS Do you know, Scott, why a reply to your note would be > setup to go just to > you and not to the list? I almost didn't catch that. I've not > noticed that > happen with any other notes on the list, even several > received today. Just > curious. :-) > > > -Original Message----- > > From: Scott Stirling [mailto:[EMAIL PROTECTED]] > > Sent: Wednesday, April 03, 2002 7:34 AM > > To: 'charles arehart '; 'JRun-Talk ' > > Subject: RE: load testing (was RE: Difficult To Reproduce > Session Issue) > > > > > > Most people have their development/staging networks are behind > > firewalls and this outside load test won't be allowed through. > > So downloading it will most likely be their only option. > > > > Just to reiterate, Microsoft Web App Stress Tool (revamped, > > called ACT, and shipped commercially in VStudio .NET > > Enterprise/Architect) is free, easy to use, and you can easily > > get it up and running in minutes and hammer the crap out of your > > site with 100 or 200 virtual users or however many you want. > > > > http://webtool.rte.microsoft.com > > > > Scott Stirling > > JRun QA > > Macromedia __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
You raise good points, Scott. Certainly, to test a site inside a firewall, you'll need to install some software. But if that's not a concern, and if you want to avoid installing (and learning) some software, this hosted solution is still a worthwhile alternative. It's dead easy. And even for just 5 users, it could be useful as Chris noted. I previously installed by WAST and OpenSTA (as well as Empirix E-test Suite, Mercury Interactive's Astra QuickTest, and Parasoft WebKing). And I found they all had complications that kept me from using them on an ongoing basis. I'm just raising this as an alternative for any who may have been through the same process. /charlie PS Do you know, Scott, why a reply to your note would be setup to go just to you and not to the list? I almost didn't catch that. I've not noticed that happen with any other notes on the list, even several received today. Just curious. :-) > -Original Message- > From: Scott Stirling [mailto:[EMAIL PROTECTED]] > Sent: Wednesday, April 03, 2002 7:34 AM > To: 'charles arehart '; 'JRun-Talk ' > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > Most people have their development/staging networks are behind > firewalls and this outside load test won't be allowed through. > So downloading it will most likely be their only option. > > Just to reiterate, Microsoft Web App Stress Tool (revamped, > called ACT, and shipped commercially in VStudio .NET > Enterprise/Architect) is free, easy to use, and you can easily > get it up and running in minutes and hammer the crap out of your > site with 100 or 200 virtual users or however many you want. > > http://webtool.rte.microsoft.com > > Scott Stirling > JRun QA > Macromedia > > -Original Message- > From: charles arehart > To: JRun-Talk > Sent: 4/2/2002 11:32 PM > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > To be honest, I was excited about OpenSTA (being open source and all), > and > the docs were really well-done, so I was psyched. But then I found I > couldn't quite get it to work at all. It was one of those justifications > for > the point Celeste had made: it didn't matter if it was free if it took a > long time to get some use of it. > > Add, too, that it did require an install of the product. One bene I > liked in > the OpenDemand tool was that it required no software install. > > /charlie > > > -Original Message- > > From: Pete Freitag [mailto:[EMAIL PROTECTED]] > > > > I like OpenSTA http://www.opensta.org/ it's a free web site load > testing > > tool. Like all testing tools you can't just sit down and use it, > > but if you > > spend a few minutes going over their example you will be on your way. > > > > > > -Original Message- > > From: charles arehart [mailto:[EMAIL PROTECTED]] > > > > I appreciate all those points, Celeste. And as someone who's also been > > around the IT world a long time (20 years), I realize that > > politics are part > > of the game. I just don't know how much that's really the heart > > of the issue > > of why people don't test. Indeed, it may be that a developer thinks > that a > > testing tool will take too long to learn to use (even if it's free) > and > > therefore it's worth even less time to try it out. > > > > But I'm trying to find out if anyone has found better alternatives. > I've > > looked at several web app testing tools (load and regression). Some > are > > free, some are expensive. I've looked at java testing tools (free and > > commercial). Yes, they all take time to learn to use, so I've been > > disinclined to use them as well. And I've talked to many of the > commercial > > companies about dropping their prices, or permitting a lower volume of > > testing for lower costs (if not free). Even testing for 5 concurrent > users > > is better than not doing any testing at all. > > > > Fortunately, I do have good news for those who've been willing to > follow > > this thread (or at least read this note): I recently learned of a > company > > (Open Demand, at www.opendemand.com) who have both a packaged load > testing > > tool and (better for many, for cost and complexity) a hosted solution > as > > well. The hosted solution is really easy to use (and no software > > to install, > > of course). In fact, after discussions some discussion I got them to > agree > > to setup a mechanism for people to trial the service for free for > > 5 virtual &
Re: load testing (was RE: Difficult To Reproduce Session Issue)
charles arehart wrote: > > ... > > I hope folks will check it out and report what you think of it. Indeed, I > hope that the 5-user test may be helpful to some in doing their first load > testing. Sure, 5 users isn't much, but if your site is relatively > low-trafficked, you ever know what running 5 concurrent users might stress > in your system. Think about it, to get 5 users at once some sites would need > to average thousands of users a day. By the same token, even if you don't > expect that load, it's also possible that you could have that load in a > sudden burst. Hey, it's free, so try it out against as many threads through > your site as you'd like. (It's not just a tool to test one page but instead > acts like a record and playback while you traverse your site creating a > "scenario".) > I agree. I think that a 5-user test can create a lot of traffic. We use Webload now (bought it), but did use an eval for a while. Five users may not sound like a lot, but remember that it is really 5 concurrent requests that you get. If you take out all the pauses that occur in real life you can get a lot of hits out of 5-user load tests. Also, even 5-user loads can show up threading/concurrency issues much better than just clicking around by yourself. Apply the 5-user load THEN click around for a while to see if anything wacky happens. So far it still hasn't cost anything! - Chris Hamlin __ This list and all House of Fusion resources hosted by CFHosting.com. The place for dependable ColdFusion Hosting. Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
To be honest, I was excited about OpenSTA (being open source and all), and the docs were really well-done, so I was psyched. But then I found I couldn't quite get it to work at all. It was one of those justifications for the point Celeste had made: it didn't matter if it was free if it took a long time to get some use of it. Add, too, that it did require an install of the product. One bene I liked in the OpenDemand tool was that it required no software install. /charlie > -Original Message- > From: Pete Freitag [mailto:[EMAIL PROTECTED]] > Sent: Wednesday, April 03, 2002 1:29 AM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > I like OpenSTA http://www.opensta.org/ it's a free web site load testing > tool. Like all testing tools you can't just sit down and use it, > but if you > spend a few minutes going over their example you will be on your way. > > _ > Pete Freitag ([EMAIL PROTECTED]) > CTO, CFDEV.COM > ColdFusion Developer Resources > http://www.cfdev.com/ > > > -Original Message- > From: charles arehart [mailto:[EMAIL PROTECTED]] > Sent: Wednesday, April 03, 2002 1:01 AM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > I appreciate all those points, Celeste. And as someone who's also been > around the IT world a long time (20 years), I realize that > politics are part > of the game. I just don't know how much that's really the heart > of the issue > of why people don't test. Indeed, it may be that a developer thinks that a > testing tool will take too long to learn to use (even if it's free) and > therefore it's worth even less time to try it out. > > But I'm trying to find out if anyone has found better alternatives. I've > looked at several web app testing tools (load and regression). Some are > free, some are expensive. I've looked at java testing tools (free and > commercial). Yes, they all take time to learn to use, so I've been > disinclined to use them as well. And I've talked to many of the commercial > companies about dropping their prices, or permitting a lower volume of > testing for lower costs (if not free). Even testing for 5 concurrent users > is better than not doing any testing at all. > > Fortunately, I do have good news for those who've been willing to follow > this thread (or at least read this note): I recently learned of a company > (Open Demand, at www.opendemand.com) who have both a packaged load testing > tool and (better for many, for cost and complexity) a hosted solution as > well. The hosted solution is really easy to use (and no software > to install, > of course). In fact, after discussions some discussion I got them to agree > to setup a mechanism for people to trial the service for free for > 5 virtual > users (with no time limit for trying it out), which is really generous of > them. I was really psyched to see that. > > If anyone here wants to try it out, the trial link is currently > http://www.opendemand.com/cf/. Don't mind the fact that the URL (and the > page) mentions CF (ColdFusion). It's just that they're initially > focusing on > the CF community but will very shortly have a link for the Java community. > Of course, the tool doesn't care what the back-end server is running. It > does the testing across the net from their hosted solution to > your site, so > the back-end's really transparent to the tool. Naturally, some may argue > that a stress test should take place within the server's network. The > company offers a packaged solution for that purpose as well. > > I hope folks will check it out and report what you think of it. Indeed, I > hope that the 5-user test may be helpful to some in doing their first load > testing. Sure, 5 users isn't much, but if your site is relatively > low-trafficked, you ever know what running 5 concurrent users might stress > in your system. Think about it, to get 5 users at once some sites > would need > to average thousands of users a day. By the same token, even if you don't > expect that load, it's also possible that you could have that load in a > sudden burst. Hey, it's free, so try it out against as many > threads through > your site as you'd like. (It's not just a tool to test one page > but instead > acts like a record and playback while you traverse your site creating a > "scenario".) > > If you're interested in trying more virtual users, the paid service starts > at $149 for 50 users for up to 60 minutes of testing (usable over whatever > period of hour
RE: load testing (was RE: Difficult To Reproduce Session Issue)
I like OpenSTA http://www.opensta.org/ it's a free web site load testing tool. Like all testing tools you can't just sit down and use it, but if you spend a few minutes going over their example you will be on your way. _ Pete Freitag ([EMAIL PROTECTED]) CTO, CFDEV.COM ColdFusion Developer Resources http://www.cfdev.com/ -Original Message- From: charles arehart [mailto:[EMAIL PROTECTED]] Sent: Wednesday, April 03, 2002 1:01 AM To: JRun-Talk Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) I appreciate all those points, Celeste. And as someone who's also been around the IT world a long time (20 years), I realize that politics are part of the game. I just don't know how much that's really the heart of the issue of why people don't test. Indeed, it may be that a developer thinks that a testing tool will take too long to learn to use (even if it's free) and therefore it's worth even less time to try it out. But I'm trying to find out if anyone has found better alternatives. I've looked at several web app testing tools (load and regression). Some are free, some are expensive. I've looked at java testing tools (free and commercial). Yes, they all take time to learn to use, so I've been disinclined to use them as well. And I've talked to many of the commercial companies about dropping their prices, or permitting a lower volume of testing for lower costs (if not free). Even testing for 5 concurrent users is better than not doing any testing at all. Fortunately, I do have good news for those who've been willing to follow this thread (or at least read this note): I recently learned of a company (Open Demand, at www.opendemand.com) who have both a packaged load testing tool and (better for many, for cost and complexity) a hosted solution as well. The hosted solution is really easy to use (and no software to install, of course). In fact, after discussions some discussion I got them to agree to setup a mechanism for people to trial the service for free for 5 virtual users (with no time limit for trying it out), which is really generous of them. I was really psyched to see that. If anyone here wants to try it out, the trial link is currently http://www.opendemand.com/cf/. Don't mind the fact that the URL (and the page) mentions CF (ColdFusion). It's just that they're initially focusing on the CF community but will very shortly have a link for the Java community. Of course, the tool doesn't care what the back-end server is running. It does the testing across the net from their hosted solution to your site, so the back-end's really transparent to the tool. Naturally, some may argue that a stress test should take place within the server's network. The company offers a packaged solution for that purpose as well. I hope folks will check it out and report what you think of it. Indeed, I hope that the 5-user test may be helpful to some in doing their first load testing. Sure, 5 users isn't much, but if your site is relatively low-trafficked, you ever know what running 5 concurrent users might stress in your system. Think about it, to get 5 users at once some sites would need to average thousands of users a day. By the same token, even if you don't expect that load, it's also possible that you could have that load in a sudden burst. Hey, it's free, so try it out against as many threads through your site as you'd like. (It's not just a tool to test one page but instead acts like a record and playback while you traverse your site creating a "scenario".) If you're interested in trying more virtual users, the paid service starts at $149 for 50 users for up to 60 minutes of testing (usable over whatever period of hours/days it takes to use up that time). Contact Don Doane ([EMAIL PROTECTED]) for more pricing info. I hope you guys will pardon if this leans towards a sales pitch. As the whole thread started out, it's really about testing, and getting people to do it more. It was just coincidental that at about the same time I was arranging them to allow this trial service. /charlie > -----Original Message- > From: Haseltine, Celeste [mailto:[EMAIL PROTECTED]] > Sent: Monday, April 01, 2002 11:11 AM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > Charles, > > My response wasn't directed at just "what to do when the bosses don't > listen", but also how to "not push your ideas to the point that > you alienate > the bosses you are trying to convince". Again, unless you are an > officer of > a company, or are the owner, you are often NOT in a "decision > making" role. > Therefore, the only thing you can do is to try and convince > people to test a >
RE: load testing (was RE: Difficult To Reproduce Session Issue)
I appreciate all those points, Celeste. And as someone who's also been around the IT world a long time (20 years), I realize that politics are part of the game. I just don't know how much that's really the heart of the issue of why people don't test. Indeed, it may be that a developer thinks that a testing tool will take too long to learn to use (even if it's free) and therefore it's worth even less time to try it out. But I'm trying to find out if anyone has found better alternatives. I've looked at several web app testing tools (load and regression). Some are free, some are expensive. I've looked at java testing tools (free and commercial). Yes, they all take time to learn to use, so I've been disinclined to use them as well. And I've talked to many of the commercial companies about dropping their prices, or permitting a lower volume of testing for lower costs (if not free). Even testing for 5 concurrent users is better than not doing any testing at all. Fortunately, I do have good news for those who've been willing to follow this thread (or at least read this note): I recently learned of a company (Open Demand, at www.opendemand.com) who have both a packaged load testing tool and (better for many, for cost and complexity) a hosted solution as well. The hosted solution is really easy to use (and no software to install, of course). In fact, after discussions some discussion I got them to agree to setup a mechanism for people to trial the service for free for 5 virtual users (with no time limit for trying it out), which is really generous of them. I was really psyched to see that. If anyone here wants to try it out, the trial link is currently http://www.opendemand.com/cf/. Don't mind the fact that the URL (and the page) mentions CF (ColdFusion). It's just that they're initially focusing on the CF community but will very shortly have a link for the Java community. Of course, the tool doesn't care what the back-end server is running. It does the testing across the net from their hosted solution to your site, so the back-end's really transparent to the tool. Naturally, some may argue that a stress test should take place within the server's network. The company offers a packaged solution for that purpose as well. I hope folks will check it out and report what you think of it. Indeed, I hope that the 5-user test may be helpful to some in doing their first load testing. Sure, 5 users isn't much, but if your site is relatively low-trafficked, you ever know what running 5 concurrent users might stress in your system. Think about it, to get 5 users at once some sites would need to average thousands of users a day. By the same token, even if you don't expect that load, it's also possible that you could have that load in a sudden burst. Hey, it's free, so try it out against as many threads through your site as you'd like. (It's not just a tool to test one page but instead acts like a record and playback while you traverse your site creating a "scenario".) If you're interested in trying more virtual users, the paid service starts at $149 for 50 users for up to 60 minutes of testing (usable over whatever period of hours/days it takes to use up that time). Contact Don Doane ([EMAIL PROTECTED]) for more pricing info. I hope you guys will pardon if this leans towards a sales pitch. As the whole thread started out, it's really about testing, and getting people to do it more. It was just coincidental that at about the same time I was arranging them to allow this trial service. /charlie > -Original Message----- > From: Haseltine, Celeste [mailto:[EMAIL PROTECTED]] > Sent: Monday, April 01, 2002 11:11 AM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > Charles, > > My response wasn't directed at just "what to do when the bosses don't > listen", but also how to "not push your ideas to the point that > you alienate > the bosses you are trying to convince". Again, unless you are an > officer of > a company, or are the owner, you are often NOT in a "decision > making" role. > Therefore, the only thing you can do is to try and convince > people to test a > product before deployment. But you can't force your bosses to do > something. > And if you try to do so, you will risk your job and your professional > reputation. > > Again, manpower issues and money issues are the key here. Even > if the tools > are free, it takes the use of "x man-hours" to test a product, > when those "x > man-hours" could be used towards something viewed as "more profitable" or > more important to the company by upper management. This is > especially true > when you are in a very sma
RE: load testing (was RE: Difficult To Reproduce Session Issue)
Drew, Playing the "uncertainty" card is defintely a very good move. As you mentioned, "higher ups" and "owners" don't like to deal with the "unknown", and sometimes this can be a good approach. Celeste -Original Message- From: Drew Falkman [mailto:[EMAIL PROTECTED]] Sent: Monday, April 01, 2002 10:39 AM To: JRun-Talk Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) I have not, quite frankly, worked in a corporate environment. But as a consultant I often face the same issues - not just with load testing, but also with general code tuning and upgrading the applications I have created. It's almost an impossible sell to convince a customer they need to spend more money and not be able to "see" the results. Celeste, this is probably old hat to you, but for others lurking, I have found some good approaches to convincing upper management that code tuning is important (these would be the same issue for load testing): 1. Track regular performance during regular traffic. Maybe take screenshots of perfmon or log performance heuristics so you can easily create graphs of performance throughout the day. IF you have SNMP monitoring tools, use them (some are pretty reasonably priced). Sometimes simply illustrating the fragility of an environment will make upper management afraid and want to know more. You could probably put together some of this preliminary stuff in a couple hours. 2. Present the above along with "what-if" scenarios (like what if a hard drive blows, or what if we get 500 users simultaneously), and explain that without load testing you simply cannot know what will happen. Most upper management types simply don't deal well with uncertainty (think of how you are always needing to give accurate assesments of the time it will take to do project X). If you illustrate that you don't know what will break first under heavy loads or what the breaking point is, that will often help. 3. The reality is that given the two points above most people will agree that load testing *would* be a good idea, but will often say that they simply don't have the time or the money to invest in it. "Let's do it next year." is not an uncommon response. The only way to get around this is to speak their language: ROI. Return on Investment. Calculating ROI, especially with something like this, is not easy and often arbitrary. But basically you need to put together the cost of doing load testing (hours + software, if necessary). Then, put together the costs for a couple scenarios. One should be the cost of the applications server going down. Estimate the amount of time it would take you to rebuild your environment. Now, add in dollars for the amount of sales lost while your server is down. Add more for the confidence lost by customers who tried to visit your site during downtime (this can be large, imagine if 100 customers never return at $100/yr/customer). Give a cost scenario during a promotion. Again, explain that this is pretty much inevitable if you don't know at what point your application will break and where. It might be good to do one that is a little less severe (people tend to think that the tragedies won't happen to them, but might believe that the server will get bogged down and you will have to spend time to troubleshoot and tune code and waste a lot of time because it is less drastic). 4. Lastly, sometimes use a promotion or current project to spur it. Simply asking what kind of traffic is expected from an upcoming promotion opens the door to you to say: "Are you sure we can handle that much?". If you follow that up fairly quickly by saying you could find out by doing some load testing will illustrate that you know what you are doing and also that you are nervous about the upcoming promotion (which should make them nervous, as well). Just some thoughts. Drew Falkman Author, JRun Web Application Construction Kit http://www.drewfalkman.com/books/0789726009/ -Original Message----- From: Haseltine, Celeste [mailto:[EMAIL PROTECTED]] Sent: Monday, April 01, 2002 8:11 AM To: JRun-Talk Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) Charles, My response wasn't directed at just "what to do when the bosses don't listen", but also how to "not push your ideas to the point that you alienate the bosses you are trying to convince". Again, unless you are an officer of a company, or are the owner, you are often NOT in a "decision making" role. Therefore, the only thing you can do is to try and convince people to test a product before deployment. But you can't force your bosses to do something. And if you try to do so, you will risk your job and your professional reputation. Again, manpower issues and money issues are the key here. Even if the tools are free, it takes the use of "x man-
RE: load testing (was RE: Difficult To Reproduce Session Issue)
I have not, quite frankly, worked in a corporate environment. But as a consultant I often face the same issues - not just with load testing, but also with general code tuning and upgrading the applications I have created. It's almost an impossible sell to convince a customer they need to spend more money and not be able to "see" the results. Celeste, this is probably old hat to you, but for others lurking, I have found some good approaches to convincing upper management that code tuning is important (these would be the same issue for load testing): 1. Track regular performance during regular traffic. Maybe take screenshots of perfmon or log performance heuristics so you can easily create graphs of performance throughout the day. IF you have SNMP monitoring tools, use them (some are pretty reasonably priced). Sometimes simply illustrating the fragility of an environment will make upper management afraid and want to know more. You could probably put together some of this preliminary stuff in a couple hours. 2. Present the above along with "what-if" scenarios (like what if a hard drive blows, or what if we get 500 users simultaneously), and explain that without load testing you simply cannot know what will happen. Most upper management types simply don't deal well with uncertainty (think of how you are always needing to give accurate assesments of the time it will take to do project X). If you illustrate that you don't know what will break first under heavy loads or what the breaking point is, that will often help. 3. The reality is that given the two points above most people will agree that load testing *would* be a good idea, but will often say that they simply don't have the time or the money to invest in it. "Let's do it next year." is not an uncommon response. The only way to get around this is to speak their language: ROI. Return on Investment. Calculating ROI, especially with something like this, is not easy and often arbitrary. But basically you need to put together the cost of doing load testing (hours + software, if necessary). Then, put together the costs for a couple scenarios. One should be the cost of the applications server going down. Estimate the amount of time it would take you to rebuild your environment. Now, add in dollars for the amount of sales lost while your server is down. Add more for the confidence lost by customers who tried to visit your site during downtime (this can be large, imagine if 100 customers never return at $100/yr/customer). Give a cost scenario during a promotion. Again, explain that this is pretty much inevitable if you don't know at what point your application will break and where. It might be good to do one that is a little less severe (people tend to think that the tragedies won't happen to them, but might believe that the server will get bogged down and you will have to spend time to troubleshoot and tune code and waste a lot of time because it is less drastic). 4. Lastly, sometimes use a promotion or current project to spur it. Simply asking what kind of traffic is expected from an upcoming promotion opens the door to you to say: "Are you sure we can handle that much?". If you follow that up fairly quickly by saying you could find out by doing some load testing will illustrate that you know what you are doing and also that you are nervous about the upcoming promotion (which should make them nervous, as well). Just some thoughts. Drew Falkman Author, JRun Web Application Construction Kit http://www.drewfalkman.com/books/0789726009/ -Original Message- From: Haseltine, Celeste [mailto:[EMAIL PROTECTED]] Sent: Monday, April 01, 2002 8:11 AM To: JRun-Talk Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) Charles, My response wasn't directed at just "what to do when the bosses don't listen", but also how to "not push your ideas to the point that you alienate the bosses you are trying to convince". Again, unless you are an officer of a company, or are the owner, you are often NOT in a "decision making" role. Therefore, the only thing you can do is to try and convince people to test a product before deployment. But you can't force your bosses to do something. And if you try to do so, you will risk your job and your professional reputation. Again, manpower issues and money issues are the key here. Even if the tools are free, it takes the use of "x man-hours" to test a product, when those "x man-hours" could be used towards something viewed as "more profitable" or more important to the company by upper management. This is especially true when you are in a very small shop as I am. I can push all I want, but if I push too much, I run the risk of alienating the very people I am trying to convince. And if I do alienate my "bosses", what have I gained for my gro
RE: load testing (was RE: Difficult To Reproduce Session Issue)
Charles, My response wasn't directed at just "what to do when the bosses don't listen", but also how to "not push your ideas to the point that you alienate the bosses you are trying to convince". Again, unless you are an officer of a company, or are the owner, you are often NOT in a "decision making" role. Therefore, the only thing you can do is to try and convince people to test a product before deployment. But you can't force your bosses to do something. And if you try to do so, you will risk your job and your professional reputation. Again, manpower issues and money issues are the key here. Even if the tools are free, it takes the use of "x man-hours" to test a product, when those "x man-hours" could be used towards something viewed as "more profitable" or more important to the company by upper management. This is especially true when you are in a very small shop as I am. I can push all I want, but if I push too much, I run the risk of alienating the very people I am trying to convince. And if I do alienate my "bosses", what have I gained for my group and "my cause"? Nothing. In fact, by pushing too far, I could end up hurting my group and my "cause" more, by not giving my management team a "graceful out" once they realize that I was probably right all along. This is called politics. And the higher you move up as an IT professional, the more important the "how well you play the political game" becomes vs. "how often you are right and your boss is wrong". Perhaps someone else on this list has a better approach to convincing their management team for taking the time and the money to do things such as testing. I'm always interested in hearing other/better approaches. But for me, this is the way I have approached all "opposition" to any IT recommendation that may be viewed as "not necessary, too costly, etc.", testing included. As I mentioned before, people and companies both make mistakes, and hopefully learn from them. Sometimes you just have to let your upper management make what you know is a decision that will probably come back to haunt them, and then just move forward from that point on. Celeste -----Original Message- From: charles arehart [mailto:[EMAIL PROTECTED]] Sent: Saturday, March 30, 2002 1:10 AM To: JRun-Talk Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) Well, those are all good thoughts, Celeste, and surely helpful to many. But my point wasn't "what do we do if the bosses don't listen" so much as "why aren't we pushing testing (especially load testing) more". I think there's a sense in many that it's not important, or it's too difficult, etc. While it's true that many sites may not get thousands (or tens of thousands) of hits in a day, it's reasonable for one getting just hundreds to get several at once, and there are clearly issues (like the one that touched off this thread) that may be triggered with just such a low level of load. If I seem to be on a bit of a bandwagon for load testing, even for "smaller shops", I am. And as has been pointed out, cost of tools isn't really the issue. I'm curious for the kinds of arguments against it that have either kept people from pursuing it or, even in the face of these assertions of the low costs and good benefits, why they would still say "nah, not going to bother". /charlie > -Original Message- > From: Haseltine, Celeste [mailto:[EMAIL PROTECTED]] > Sent: Friday, March 29, 2002 10:55 AM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > Charlie, > > I just try to point out the business cases for testing. With many > owners/managers, it comes down to money and time, with time also being > related to money. As a professional, I ALWAYS write up my > business case in > a 1 page summary, which hits the highlights as to why ANY > software product, > internet or otherwise, should be tested. __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
Well, those are all good thoughts, Celeste, and surely helpful to many. But my point wasn't "what do we do if the bosses don't listen" so much as "why aren't we pushing testing (especially load testing) more". I think there's a sense in many that it's not important, or it's too difficult, etc. While it's true that many sites may not get thousands (or tens of thousands) of hits in a day, it's reasonable for one getting just hundreds to get several at once, and there are clearly issues (like the one that touched off this thread) that may be triggered with just such a low level of load. If I seem to be on a bit of a bandwagon for load testing, even for "smaller shops", I am. And as has been pointed out, cost of tools isn't really the issue. I'm curious for the kinds of arguments against it that have either kept people from pursuing it or, even in the face of these assertions of the low costs and good benefits, why they would still say "nah, not going to bother". /charlie > -Original Message- > From: Haseltine, Celeste [mailto:[EMAIL PROTECTED]] > Sent: Friday, March 29, 2002 10:55 AM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > Charlie, > > I just try to point out the business cases for testing. With many > owners/managers, it comes down to money and time, with time also being > related to money. As a professional, I ALWAYS write up my > business case in > a 1 page summary, which hits the highlights as to why ANY > software product, > internet or otherwise, should be tested. __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
Charlie, I just try to point out the business cases for testing. With many owners/managers, it comes down to money and time, with time also being related to money. As a professional, I ALWAYS write up my business case in a 1 page summary, which hits the highlights as to why ANY software product, internet or otherwise, should be tested. Since I am a Texas Registered Professional Engineer, I will sometimes also sign and seal the document with my license number (depending on how big the financial/legal liability could be to my client and/or to me), although this is NOT a requirement in TX. This gives the document a very formal and professional look/feel. It also covers my "rear-end" in the event someone wants to try and place the blame on me. I take the position that some of my friends who are in the legal profession take. My job is to inform my client/management of any potential pitfalls/liabilities/impacts to the company, and provide them with a recommended course(s) of action(s). Whether they accept and act on my advice is up to them. They are the decision makers, not me. If not, then I have clearly document what could happen. That doesn't mean that they cannot try to pass the blame back onto me when something does go wrong. But if they do, I can always refer back to my original document dated "such and such", and remind them that the final decision was of their making. I then move the conversation forward to the "now this is what we need to do to get this fixed" arguments. It's never in your best interest, or very professional, to "rub it in" to a client when they do get "bitten" by a poor decision. We all learn through our mistakes, and businesses and owners are no different. The big thing here to note is documentation. Always document your objections, concerns, etc., and do it in a factual and professional manner. Never let any emotion "show through" in your correspondence. The person you may be dealing with might be a "jerk" in real life, but it doesn't do you any good to remind him/her of that fact. Your objective is to protect your client/company using your skills sets/education/experience (and in some cases to try and win them over), and to do it in a non-threatening and professional manner. After a while, when people begin to realize that 9 times out of 10 your right, they will begin to start listening to you and to take notice of you. And taking such a professional course of action will gain you a positive reputation with your co-workers and within the developer community. Don't know if this advice helps, but I have found over the years this is best course of action. Particularly for a women in a male dominated industry (which I am). Just remember you are being paid to do a job to the best of your abilities, and part of that job is to "warn" your client/employer of any potential pitfalls their actions/decisions could have on their business/company. Unless you are an owner, or an officer of the company, the final decision is probably not yours to make, so don't take the "rejection" of your recommendations/advice personally. It's just business. Just make sure you document everything to protect yourself and your professional reputation. Celeste -Original Message----- From: charles arehart [mailto:[EMAIL PROTECTED]] Sent: Thursday, March 28, 2002 11:47 PM To: JRun-Talk Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) Yep, Dave and Ted have ID'ed the two most popular free tools I know of. So cost isn't really the issue, once one knows of these. It's more just a matter of deciding that the effort's worthwhile. Sounds like in your case, Celeste, that the bosses really just aren't motivated to worry about testing. That's the part I want to address. What do we, as professionals, do (and know, and say) to promote more effective testing (load and otherwise). What's really keeping it from being more prominent? Just some rhetorical questions. If there are no replies, I'll know it's the sound of one hand clapping. :-) /charlie > -Original Message- > From: Dave Watts [mailto:[EMAIL PROTECTED]] > Sent: Thursday, March 28, 2002 5:06 PM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > > If you do find other freebie testing tools, I would be > > interested in the names/links. > > Take a look at OpenSTA: > http://www.opensta.org/ > > It's open-source, and free, and looks pretty good. I haven't used > it for an > actual testing engagement yet (I'm used to SilkPerformer) but it > looks like > it covers all the bases pretty well. > > Dave Watts, CTO, Fig Leaf Software > http://www.figleaf.com/ > voice: (202) 797-5496 > fax: (202) 797-5444 &g
RE: load testing (was RE: Difficult To Reproduce Session Issue)
Yep, Dave and Ted have ID'ed the two most popular free tools I know of. So cost isn't really the issue, once one knows of these. It's more just a matter of deciding that the effort's worthwhile. Sounds like in your case, Celeste, that the bosses really just aren't motivated to worry about testing. That's the part I want to address. What do we, as professionals, do (and know, and say) to promote more effective testing (load and otherwise). What's really keeping it from being more prominent? Just some rhetorical questions. If there are no replies, I'll know it's the sound of one hand clapping. :-) /charlie > -Original Message- > From: Dave Watts [mailto:[EMAIL PROTECTED]] > Sent: Thursday, March 28, 2002 5:06 PM > To: JRun-Talk > Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) > > > > If you do find other freebie testing tools, I would be > > interested in the names/links. > > Take a look at OpenSTA: > http://www.opensta.org/ > > It's open-source, and free, and looks pretty good. I haven't used > it for an > actual testing engagement yet (I'm used to SilkPerformer) but it > looks like > it covers all the bases pretty well. > > Dave Watts, CTO, Fig Leaf Software > http://www.figleaf.com/ > voice: (202) 797-5496 > fax: (202) 797-5444 > > __ This list and all House of Fusion resources hosted by CFHosting.com. The place for dependable ColdFusion Hosting. Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
> If you do find other freebie testing tools, I would be > interested in the names/links. Take a look at OpenSTA: http://www.opensta.org/ It's open-source, and free, and looks pretty good. I haven't used it for an actual testing engagement yet (I'm used to SilkPerformer) but it looks like it covers all the bases pretty well. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ voice: (202) 797-5496 fax: (202) 797-5444 __ This list and all House of Fusion resources hosted by CFHosting.com. The place for dependable ColdFusion Hosting. Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
Celeste, Microsoft has a stress test tool that you can download for free. See the following KB article: http://www.macromedia.com/v1/Handlers/index.cfm?ID=21951&Method=Full Ted Zimmerman -Original Message- From: Haseltine, Celeste [mailto:[EMAIL PROTECTED]] Sent: Thursday, March 28, 2002 4:15 PM To: JRun-Talk Subject: RE: load testing (was RE: Difficult To Reproduce Session Issue) Charlie, I would love to use some load testing tools, and I have gone to the mat with both the owner and my boss (the VP) to try and purchase some. But you have to keep in mind that most companies are really on tight budgets right now, and some smaller IT groups such as this one are having to make do with either free testing tools/shareware that is out on the internet, or open source tools via Sun/Jakarta. The company I am currently working with is a small manufacturer, who has a very narrow profit margin on the products he produces. He is in a niche market, but because he has such a narrow profit margin, and because the owner is close to retirement (he is 60), he is extremely reluctant to spend any money. In fact, the only reason he hired me was that some of his biggest clients told him that if he did not institute a web ordering system, they would consider taking their business to his competition. Even though the owner really dislikes computers, the threat by his larger clients caught his attention and he is now being "forced" into the "web age" kicking and screaming all the way. Bottom line, both he and the VP say they want to take their chances on when the site would go down and under what load conditions. I've tried to convince them that when your revenue is being generated via a web site, this is NOT the way you want to find coding errors and/or other problems, but I have not been successful in that endeavor yet. They feel that most of their customers will not actually use the site, and they do not intend to advertise it's existing in order to keep the traffic down. The concept that someone might just "spam" their site for the hell of it, and cause the site to crash, doesn't concern at this time. If you do find other freebie testing tools, I would be interested in the names/links. Celeste -Original Message- From: charles arehart [mailto:[EMAIL PROTECTED]] Sent: Thursday, March 28, 2002 12:53 PM To: JRun-Talk Subject: load testing (was RE: Difficult To Reproduce Session Issue) Celeste, you say: > We haven't seen this problem, not yet anyway. But our servers > are not under > much of a load yet, as our site is just getting off the ground. We expect > that to change over the next three to four months. If we do run into the > same problem, I suspect that will be the final straw for the owner of this > facility, and he too will decide to go with another JSP/Servlet > server. I'm curious if you're considering load testing before just waiting to see what happens. Indeed, I'd like to open this discussion to a broader one of whether people do use load testing tools, especially if they do anticipate a growth in load. Or is the feeling that they're too expensive or cumbersome. Perhaps people can share their experiences with any load testing tools, indicating as well if the relative costs and challenges in using them. Perhaps they can also clarify their own load requirements: clearly someone with a commercial, clustered site with hundreds of thousands of hits per day will have different needs (and preferences) over someone with a smaller organization (or intranet) site getting hundreds of hits per day. Still, for issues like this one being discussed, just simulating many per second in a load testing tool could identify a problem that might otherwise not crop up until later. As Celeste says, better to know about it sooner than later. A testing tool seems of more value to people than they seem to give it credit. What are the facets that stop people from using them? Cost? Complexity? Lack of awareness? Choosing among many alternatives? Hopefully this thread will at least help us on the list get a better idea of what options make sense for our varying needs, and we can pass along our discoveries to others, and so on. /charlie __ This list and all House of Fusion resources hosted by CFHosting.com. The place for dependable ColdFusion Hosting. Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: load testing (was RE: Difficult To Reproduce Session Issue)
Charlie, I would love to use some load testing tools, and I have gone to the mat with both the owner and my boss (the VP) to try and purchase some. But you have to keep in mind that most companies are really on tight budgets right now, and some smaller IT groups such as this one are having to make do with either free testing tools/shareware that is out on the internet, or open source tools via Sun/Jakarta. The company I am currently working with is a small manufacturer, who has a very narrow profit margin on the products he produces. He is in a niche market, but because he has such a narrow profit margin, and because the owner is close to retirement (he is 60), he is extremely reluctant to spend any money. In fact, the only reason he hired me was that some of his biggest clients told him that if he did not institute a web ordering system, they would consider taking their business to his competition. Even though the owner really dislikes computers, the threat by his larger clients caught his attention and he is now being "forced" into the "web age" kicking and screaming all the way. Bottom line, both he and the VP say they want to take their chances on when the site would go down and under what load conditions. I've tried to convince them that when your revenue is being generated via a web site, this is NOT the way you want to find coding errors and/or other problems, but I have not been successful in that endeavor yet. They feel that most of their customers will not actually use the site, and they do not intend to advertise it's existing in order to keep the traffic down. The concept that someone might just "spam" their site for the hell of it, and cause the site to crash, doesn't concern at this time. If you do find other freebie testing tools, I would be interested in the names/links. Celeste -Original Message- From: charles arehart [mailto:[EMAIL PROTECTED]] Sent: Thursday, March 28, 2002 12:53 PM To: JRun-Talk Subject: load testing (was RE: Difficult To Reproduce Session Issue) Celeste, you say: > We haven't seen this problem, not yet anyway. But our servers > are not under > much of a load yet, as our site is just getting off the ground. We expect > that to change over the next three to four months. If we do run into the > same problem, I suspect that will be the final straw for the owner of this > facility, and he too will decide to go with another JSP/Servlet > server. I'm curious if you're considering load testing before just waiting to see what happens. Indeed, I'd like to open this discussion to a broader one of whether people do use load testing tools, especially if they do anticipate a growth in load. Or is the feeling that they're too expensive or cumbersome. Perhaps people can share their experiences with any load testing tools, indicating as well if the relative costs and challenges in using them. Perhaps they can also clarify their own load requirements: clearly someone with a commercial, clustered site with hundreds of thousands of hits per day will have different needs (and preferences) over someone with a smaller organization (or intranet) site getting hundreds of hits per day. Still, for issues like this one being discussed, just simulating many per second in a load testing tool could identify a problem that might otherwise not crop up until later. As Celeste says, better to know about it sooner than later. A testing tool seems of more value to people than they seem to give it credit. What are the facets that stop people from using them? Cost? Complexity? Lack of awareness? Choosing among many alternatives? Hopefully this thread will at least help us on the list get a better idea of what options make sense for our varying needs, and we can pass along our discoveries to others, and so on. /charlie __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
load testing (was RE: Difficult To Reproduce Session Issue)
Celeste, you say: > We haven't seen this problem, not yet anyway. But our servers > are not under > much of a load yet, as our site is just getting off the ground. We expect > that to change over the next three to four months. If we do run into the > same problem, I suspect that will be the final straw for the owner of this > facility, and he too will decide to go with another JSP/Servlet > server. I'm curious if you're considering load testing before just waiting to see what happens. Indeed, I'd like to open this discussion to a broader one of whether people do use load testing tools, especially if they do anticipate a growth in load. Or is the feeling that they're too expensive or cumbersome. Perhaps people can share their experiences with any load testing tools, indicating as well if the relative costs and challenges in using them. Perhaps they can also clarify their own load requirements: clearly someone with a commercial, clustered site with hundreds of thousands of hits per day will have different needs (and preferences) over someone with a smaller organization (or intranet) site getting hundreds of hits per day. Still, for issues like this one being discussed, just simulating many per second in a load testing tool could identify a problem that might otherwise not crop up until later. As Celeste says, better to know about it sooner than later. A testing tool seems of more value to people than they seem to give it credit. What are the facets that stop people from using them? Cost? Complexity? Lack of awareness? Choosing among many alternatives? Hopefully this thread will at least help us on the list get a better idea of what options make sense for our varying needs, and we can pass along our discoveries to others, and so on. /charlie __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm Archives: http://www.mail-archive.com/jrun-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
