On Wed, Jan 1, 2014 at 8:04 PM, sebb <[email protected]> wrote: > On 1 January 2014 15:08, Philippe Mouawad <[email protected]> > wrote: > > On Tue, Dec 31, 2013 at 9:00 PM, Vladimir Sitnikov < > > [email protected]> wrote: > > > >> >> I believe JMeter plugin-like interface between injector and > performance > >> >> repository would enable live monitoring of sytem under test. > >> sebb>Exactly. The JMeter architecture is designed to allow for easy > >> integration of 3rd party plugins. > >> > >> I need to double-check, however I did not manage to specify "path to > >> plugins.jar folder" via command line option. > >> Ultimately I would love to have JMeter installation and plugins in > >> completely separate folders. That simplifies "jmeter upgrade", "check > >> what plugins are installed", "compose test harness from maven" > >> usecases. > >> > >> Add this to user.properties: > > plugin_dependency_paths which will contains the path to third party jar > > > > user.classpath which will contain path to dependencies of third party jar > > > > > > > >> Are there "asynchronous output" interfaces from JMeter? > >> > > > > in remote testing you are in async mode with: > > mode=StrippedAsynch > > or > > mode= Asynch > > > >> > >> Is there a way to send listener result via regular samplers? > >> > > > > Why would you like to do that ? > > > > sebb> Proper analysis should take place offline after the test has > >> completed. > >> Very true. > >> However, it is quite important to perform online analysis to be able > >> to adjust the test. > >> Say, adjust the load, fix bugs in script, correct system configuration, > >> etc. > >> > >> >> One can parse raw csv/xml results and upload for the analysis, > however > >> it > >> >> is likely to create big latency gap between collection and the > >> >> visualization. > >> > > >> sebb> Is that really a problem for most users? > >> How do we measure that? > >> > > > > A vote would be a way to sort it out. > > Perhaps, though it's very difficult to know whether a vote is > representative. > > >> > >> Here are the most relevant scenarios for our company: > >> 1) Durability testing. Say, you launch 1..7days long test script. > >> It is crucial to know if the system is yet stable. > >> That includes multiple KPIs (throughput, latency, failrate%, memory > >> consumption, gc, cpu%), and the request processing KPIs are not the > >> least important ones. > >> In case JMeter sampler info is not ready until the whole test is > >> finished it is major drawback. > >> > > > > Agree > > There are ways round this. > The Generate Summary Results Listener was designed for just this sort > of scenario. > I was involved in testing a large system where the test ran for > several days, and that was ideal for checking that the test was still > running well. > > Obviously it only provides basic info, but sufficient to check that > the test is generally behaving (or not). > Its big advantage is the low overhead and simplicity. > > > > >> > >> 2) Automated scalability testing. Say you want to identify maximum > >> load the system will sustain. One way to identify it is to gradually > >> increase the load and see if the system is stable (e.g. queues do not > >> build up, failrate is 0, response times are stable, etc). > >> Having data displayed in near-realtime helps a lot. > >> Especially, when you run test suite in different environment (e.g. > >> acceptance of new software/hardware) > >> > > > > Agree > > We used the Generate Summary Results Listener for that as well. > > >> > >> 3) Tight-schedule testing. When performing load testing at customer > >> environment (e.g. acceptance of production environment), it is > >> important to make informed decisions. Is good to see if your test > >> works as expected when the test is running, not when you've done 4 > >> hours of testing and analyzing afterwards. > >> > > > > Agree > > > >> > >> 4) Red-green detection during regular 30-60min testing. Our scenarios > >> involve multiple jmx scripts and lots of samples. > >> We use JMeter GUI only for script development. We use just console for > >> load testing to avoid injector slowness/out of memory/etc. > >> Currently it is hard to tell if the test goes as expected: failrate%, > >> number of samples, response times, etc. > > Most of that is available from the Generate Summary Results Listener > already. > > >> > >> Agree > > > > > >> sebb> That would be a sensible addition to JMeter to allow performance > data > >> > to be readily saved to an arbitrary repo. > >> True. > >> OOB samplers might work (e.g. http sampler) might work for the "output > >> interfaces". > >> For instance "http sampler" under "stats listener" posts the result of > >> "stats listener" to the configured end-point. > >> > >> However, I believe this kind of integration should be asynchronous to > >> avoid impact result collection on the test scenario. If trying to use > >> sync result posting, we could result into multi-second test hang due > >> to lag of performance repository receiver. > >> > >> Currently the Graphite Listener patch it is implemented this way: > > 1) Synchronously, success , failures, min and max are updated (very fast > > computations), percentile related data is also computed (Looking at code > of > > commons-math3 I don't see why it would not perform fine). If further > tests > > show the percentile computation is costly , then we could delay this to > the > > 2) thread below. But the tests I did did not show problem. > > 2) And in the background, a task runs every minute to send current > computed > > results to Graphite server , resetting datas except for percentile > related > > ones. > > Seems to me all this could be done by having a generic interface to > enable the raw results to be saved to whatever backend is required. >
We could refactor code in the following way: - GraphiteListener => BackendListener - Introduce Backend Interface - Backend would be called in testStarted/testEnded and within the existing run() to send metrics - There would be a select box to select backend. Backend implementations would be searched within classpaths - Graphite part and PickleMetricsManager would be moved to Graphite Backend. We could after that add a JBDC Backend. > > What I don't agree with is yet more code that is very specific to a > single 3rd party software solution. > > Whatever is added needs to be > - generally useful to the JMeter population. Niche requirements should > be met by 3rd party add-ons developed for the purpose. > - use a generic API, for example like JSR223, JDBC etc. > > I understand your wish but take the history of JMeter. At some time there was a need for custom scripting but no standard like BSF of JSR223. So you selected Beanshell . Once standard was introduced we moved to it. I think we cannot wait for all standards to exist, for example if we take NoSQL Databases, it is hard to tell when a JDBC equivalent will be available, does it means we should wait until they are available ? I don't think so. > > > > > >> Regards, > >> Vladimir Sitnikov > >> > > > > > > > > -- > > Cordialement. > > Philippe Mouawad. > -- Cordialement. Philippe Mouawad.
