These are all good notes, and I was certainly thinking about Ben's tests when I wrote my initial email.
The scope of this can expand dramatically. What I really want, for a first pass, is an answer to the question: Is Tapestry 5.2 faster than 5.1? Does it use less memory? On Thu, Sep 2, 2010 at 11:33 PM, Kalle Korhonen <kalle.o.korho...@gmail.com> wrote: > Ben, what you are talking about seems to be geared more towards > testing readiness and scalability of a production application. For > benchmarking multiple versions of the framework, I'd assume data > access would never come to play as there should be no database > whatsoever. Maintaining a fully synthetic test should also be far > easier than maintaining a scalability test for a real world > application. Also, standardized, guaranteed QoS VMs are great for > scalability testing when you are mostly looking for "good enough" as > an answer but if you are really comparing versions of the framework > over time, the differences should be measured in milliseconds even > with thousands of repeated executions, so it's critical that extra > variables are reduced to minimum just the same way as if we > benchmarked specific features of JVM against different versions. On > the other hand, creating and benchmarking a close-to-real-world > blueprint application is always good marketing, which Tapestry > certainly needs. > > Howard expressed a desire for doing both type of testing, but I don't > think its a good idea to try and combine the different goals into the > same test suite. In a fully synthetic test, the absolute numbers don't > matter, only the differences do. And you certainly couldn't compare > results of a synthetic test implemented in one framework to test > results implemented in another framework, but only to different > versions of itself. Seems that the first thing to do is to decide what > exactly we want to measure. > > Kalle > > > On Thu, Sep 2, 2010 at 10:55 PM, Ben Gidley <b...@gidley.co.uk> wrote: >> I did a comparative load test >> http://blog.gidley.co.uk/2009/05/tapestry-load-testing-round-up.html a while >> back and we do a lot of load testing for SeeSaw.com (mainly to compare >> versions). I did create a 'comparative' struts application for that >> evaluation (source code >> http://sites.google.com/a/gidley.co.uk/tapestryloadtest/Home/test-5) >> >> We use Grinder as our load test tool - as it handles clustered load >> generating agents well and the Jython scripting makes it easy to share steps >> between 'journeys' in the application. You probably don't need to worry >> about clustering - as it isn't a core tapestry 'feature' (rather an >> environmental one), but even without that feature grinder is easy to work >> and powerful. >> >> If your goal is to test tapestry the first issue you will hit is any Data >> Access - in our apps the data access piece is 90% of the page time. In >> SeeSaw we 'solve' this by heavily caching data so most pages only access >> data from a Cache. I would suggest to get a test that focused on tapestry >> you load you data from in memory caches. That way you won't be measuring how >> fast Hibernate or your DB(etc) performs. >> >> Another big issue we hit is breaking/maintaining our load test scripts - as >> they effectively reverse engineer some tapestry behaviours (especially AJAX >> ones) they can get broken quite easily. Our solution is to run them single >> threaded as part of our CI builds. This usually catches obvious changes. >> >> Another thing to watch out for is your app server. Jetty is an excellent >> high load container - but depending on how it is configured it can >> 'throttle' throughput (this is excellent thing in production). This can skew >> your test as you end up measuring the throttling and not the application. >> The trick is to set the thread pools in Jetty far higher than you would in >> production so you can be sure you are not maximising them. >> >> I would advise trying to use controlled hardware for your test - slight >> environmental differences can totally adjust the results. EC2 and its like >> are popular for this - do Apache have anything similar? I can get you time >> on (but not direct access to) the ioko cloud (which is a VMWare VSphere >> cluster), which does let you nearly guarantee QOS. >> >> It may be a good idea to build the test as something that can be easily >> deployed as a Virtual machine as it would let you easily deploy it anywhere. >> I have previously used Suse Studio (http://susestudio.com/) to build a very >> small VM with a maven script on it that updates/runs my app. The VM's built >> there will run pretty much anywhere. >> >> To find issues with a load test I recommend YourKit. It gives you a ton of >> monitoring and via its 'probes' you can see right inside tapestry. It is >> free for open source projects. I would also recommend enabling JMX in your >> application and periodically writing the stats somewhere. You can do this >> simply with VisualVM or you can get more advanced and store them in a >> database and graph them - ioko use Cacti http://www.cacti.net/ and >> http://jmxmonitor.sourceforge.net/ (though it may be a bit more complex for >> what you want). >> >> >> Ben Gidley >> >> On Thu, Sep 2, 2010 at 11:07 PM, Thiago H. de Paula Figueiredo < >> thiag...@gmail.com> wrote: >> >>> On Thu, 02 Sep 2010 18:30:49 -0300, Howard Lewis Ship <hls...@gmail.com> >>> wrote: >>> >>> One of my long term and unrealized goals for Tapestry is to have a >>>> legitimate performance testing lab. >>>> >>> >>> Cool! >>> >>> >>> I'd like the ability to run a "standard" application through its >>>> paces, collect statistics, and see how well each new Tapestry release >>>> compares to its predecessor in terms of general performance: response >>>> time, memory utilization, saturation and recovery. Ideally, it would >>>> be nice to create equivalent JSP/Spring MVC/Struts/Wicket >>>> implementations of the same app. >>>> >>> >>> I've done something a little similar, but in a smaller scope, for a >>> presentation comparing Struts 1, Struts 2, JSF and Tapestry 5. I implemented >>> a simple application, DAOs and simple business rules layer, and them >>> implemented its web interface using each of the frameworks. >>> >>> >>> I'm really concerned with ensuring that 5.2 out-performs 5.1, and uses >>>> less memory. The singleton pages approach should lower the memory >>>> utilization ... but I'm concerned that the new AOP stuff inside >>>> ClassTransformation (which has a tendency to create lots of little >>>> one-off classes) will overshadow the other improvements. >>>> >>> >>> Someone in the mailing list mentioned Parfait ( >>> http://code.google.com/p/parfait/wiki/IntroductionToParfait). It could be >>> helpful for that. >>> >>> -- >>> Thiago H. de Paula Figueiredo >>> Independent Java, Apache Tapestry 5 and Hibernate consultant, developer, >>> and instructor >>> Owner, Ars Machina Tecnologia da Informação Ltda. >>> http://www.arsmachina.com.br >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@tapestry.apache.org >>> For additional commands, e-mail: dev-h...@tapestry.apache.org >>> >>> >> >> >> -- >> Ben Gidley >> >> www.gidley.co.uk >> b...@gidley.co.uk >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@tapestry.apache.org > For additional commands, e-mail: dev-h...@tapestry.apache.org > > -- Howard M. Lewis Ship Creator of Apache Tapestry The source for Tapestry training, mentoring and support. Contact me to learn how I can get you up and productive in Tapestry fast! (971) 678-5210 http://howardlewisship.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tapestry.apache.org For additional commands, e-mail: dev-h...@tapestry.apache.org
