Here are my $0.02: - lack of dedicated technical support (unlike using commercial tools) - unable to integrate with other (commercial) tools (e.g., IBM's "Jazz")
Otherwise, jmeter appears to me as a good tool (and it's free). For me, the first item is the thing that bugs me the most. I'm not a proficient user of jmeter and end up spending a lot of time looking at postings about how to do things; a compilation of examples, ranging from trivial to very elaborate, would be nice. For example, an idiot-proof example on how to pass parameters between threads would be nice. I don't bother with jmeter's graphing - I dump the output to a .csv file and then read it using (commercial) data mining tools to do analysis much more sophisticated than jmeter is capable of doing (e.g., paired t-tests, cross-correlations, distribution fits, etc.). Hence, I cannot comment on how useful the graphing is to me. Overall, it's a good tool. Hope this helps, Bo -----Original Message----- From: David Luu [mailto:[email protected]] Sent: Monday, October 15, 2012 8:26 PM To: JMeter Users List Subject: Some assumptions on JMeter limitations. Your thoughts? Hello, I was asking for feedback on our usage of JMeter within my organization and got some responses with some disadvantages/limitations of JMeter. Based on the responses, most of it to me seems like some assumptions made w/o knowing/researching the full capabilities of JMeter. It took me some time to explore and find out JMeter's feature set. But in any case thought I'd ask the community for input regarding these (assumed?) disadvantages, extracted snippet of feedback: "I don't expect us to use JMeter long-term because it is pretty difficult to: * extract and re-use components, e.g. sign-in * verify or review the test or your changes to it are doing what you think it is doing; you're stuck with either understanding the test through the UI or reading jmx directly Perhaps you can share some insights in how you have addressed the above points. I think one way to address these issues is to express our performance/load tests in a real programming environment. This would give us the normal power we expect to manage our test software. There are a few systems that allow for this, e.g. gatling [1], grinder [2], iago [3] [1] http://gatling-tool.org/ [2] http://grinder.sourceforge.net/ [3] http://twitter.github.com/iago/ ... Also, having a programming language support gives you a better control on the load testing scripts, but Jmeter doesn't have that." The re-use issue seems interesting to me. How best to do that in JMeter? For me all I could think of is building a standard test template with common components and settings and everyone works off a copy of that customizing as needed. But is there a better way to reuse-share a component like HTTP header manager, or User Parameters or User Defined Variables, or Regular Expression Extractor, etc. with the customized variables, values, patterns w/o having to copy paste from one test plan to another? On the understanding/reading JMeter test plan, my only thought to that was good test plan design in terms of renaming components in test plan, labelling them with description of what that component is doing in the test and adding in info in the comment fields for components that have them. And that would apply to both reviewing test plan in GUI and from parsing JMX file. Any thoughts beyond that? For the programmatic functionality area, I know we can get that by using JMeter custom plugins, or import Java libraries for use with JUnit/BSF/Beanshell pre/post processors, assertions, and samplers. Build out the needed functionality in code & call it from JMeter, using JMeter for threading, concurrency, load management, and possibly reporting/logging too. But any thoughts beyond that? I'm assuming my colleagues were looking more for a full featured IDE type tool or language binding based framework like Selenium 2/WebDriver where you write in code using "JMeter" APIs to do the load testing. Regards, David --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
