Hello Milamber, My responses below. Regards Philippe On Mon, May 21, 2012 at 11:51 PM, Milamber <[email protected]> wrote:
> > > Le 21/05/2012 22:12, Philippe Mouawad a ecrit : > > Hello, > > I read recently this little comparison : > > https://github.com/excilys/gatling/wiki/Benchmarks< > https://github.com/excilys/gatling/wiki/Benchmark> > > > > I reviewed the test plan that was used to make the test. > > It seems to me the test is little biased: > > > > - View Results Tree is in test plan (as it uses a lot of memory, it's > a > > big issue) > > - View Results in Table (same thing) > > - 3 Non Standard JMeter listener, so It's not pure JMeter: > > - jp@gc - Transactions per Second > > - jp@gc - Response Times Over Time > > - jp@gc - Active Threads Over Time > > - Default XML output seems to have been used , it's against best > > practices > > > > > > Besides the following conclusions that seem to me non scientific and > purely > > subjective: > > > > - The testers, new to both Gatling and JMeter found that JMeter was > > harder to learn and use than Gatling to create the simulations, > despite the > > use of a proxy. > > > > > > I will only look at other ideas expressed: > > > > - JMeter creates one thread per user simulated. If there is not enough > > memory allocated to the JVM, it can crash trying to create these > threads > > - This need to be detailed, cause either it fails with OOM and it's > > not during thread creation, either it fails with "unable to create > new > > native thread" > > - For instance, JMeter could not run 1500 users with 512 MB (what was > > used for Gatling even with 2000 users); OutOfMemoryErrors are > recorded in > > the table as *OOM* > > - => I made a Test with up to 2000 Threads with 512 m without any > > crash, it depends on Test and on application > > - Another problem occurred with the 2000 users simulations; it seems > > that JMeter can not simulate more than 1514 users independently from > the > > memory that was allocated to the JVM > > - => I made a Test with up to 2000 Threads with 512 m without any > > crash, so assertion is false, it depends on Test and on application > > > > > > As it's difficult to install the application used for test (last version > > does not seem to work as expected) , if they provide a working WAR > against > > a local Postgres DB I will be happy to test with it. > > But in current state, application seems to be packaged for cloud or H2 > > local DB, I didn't want to spend too much time setting up application as > I > > don't know its real status. > > > > I just tried to run Test Plan against a blank tomcat to verify what they > > say about Thread Creation, I didn't find any issue on this. > > > > So I decided to make a very simple scenario test on Tomcat Examples (It > > goes to Session Example, adds attribute, go back to index, go back to > > Session Example, test contains Response assertion for each Request). > > > > Please, indicate the Tomcat version used? it's the same machine? config > of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM editor? > > Tomcat apache-tomcat-6.0.24 -Xms256m -Xmx1024m <Connector port="8080" protocol="HTTP/1.1" compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json" compression="off" socketBuffer="8" maxThreads="400" connectionTimeout="20000" redirectPort="8443" /> Set session timeout in web.xml to 1min java version "1.6.0_29" Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527) Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode) Mac OS 10.6.8 JMeter and Tomcat on same machine No particular OS Tuning 8GO RAM No swap, nothing runnning except these 2 Tomcat CPU around 5% Memory around 50 mo JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)? > > Same config No TCP tuning > Can you post the jmx file? > > Will it be accepted on this list ? > Milamber > > > > It is not at all representative but it is a way for me to check potential > > issues in JMeter and performance changes accross versions. > > > > I ran the test with 1500 VU using JMeter 2.5.1, 2.7 (current trunk) with > > -Xmx512m, 10 minutes run and CSV output against a local Tomcat (I restard > > tomcat between tests and control its health): > > > > - I noticed that current trunk version behaves much better in terms of > > memory than 2.5.1 or 2.6: > > - In 2.5.1 : > > - GC activity is much higher with around 5 GC CPU peaks every 2 > > minutes, and 20 FULL GC of 700 to 800 ms each > > - Throughput: 97,71% > > - Pauses : 13,69s > > - Mém : 391M/min > > - Full GC tend to be much more frequent at end of test > > - 2.7: > > - no GC CPU peak, 1 FULL GC > > - Throughput:98.54 > > - Pauses : 8.9s > > - 1108m /min > > - Summary: > > - 25.1: > > - 164676 samples in 605,1s > > - 272,2/s > > - Avg: 97 > > - 2.7: > > - 165367 in 605.0s > > - 273.3/s > > - Avg: 228 > > - I also noticed results have a much better look: > > - in 2.5.1, Transactions/s are around 300 / sec during 4 minutes, > > then drop to 200/s, go up to 400/s , then down to 260/s and > > finally 200/sec > > - in 2.7, Transactions stay around 300/sec > > > > > > > > Actions: > > > > - I think it would be useful to have some reference application on > which > > we could test JMeter behaviour > > - What would the best place to put these comparisons ? wiki ? which > > indicators should we put ? > > - Further testing should be done against a richer application that > could > > be deployed locally > > > > > > Regards > > Philippe > > > > > > -- Cordialement. Philippe Mouawad.
