-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ole Ersoy wrote: > Hey Alex,
Hey Ole good to hear from you again. > Welcome back from vacation! Thanks. It was too short. > - Created maven archetypes specific to performance > testing directory servers, with a set of directories > for DS configuration files. Hmmm I see where you're going. You want to generate projects that have a server.xml that can be customized for the test case. What about having test cases for embedded operation where the server.xml does not come into play? Thus there should be one > archetype per DS. So apache would have one, and > OpenLDAP would have another. Ok I see now. The OpenLDAP archetype is a bit more difficult to pull off tho since it is C-code and not embeddable or launchable via test cases. We might have to fork an instance of it. The configuration files > directory should mirror the DS's configuration file > directory structure. This isolates the DS's > configuration settings per performance test, and > provides the initial baseline for everyone to use. So by using an archetype you plan to isolate each performance test case into a separate maven module? > - Create another Archetype input directory for the > LDIF files. Ok starting to loose you here. Perhaps my knowledge of maven archetypes are breaking down now. Can you elaborate? > - Write a mojo that embeds ApacheDS, loads the > configuration files, and then runs all the tests on > the specific server instance, and then generates > reports. Yeah this I understand. We can do this pretty easily. This way each instance of this archetype is > running the DS in an isolated manner. Hmmm again loosing you here on the meaning or use of the m2 archetype. Would also be > cool to integrate this stuff with Eclipse TPTP, using > something similar to the maven eclipse plugin...that > way all the results are cross compatible with TPTP and > can be opened up in an eclipse TPTP project...maybe in > version 2 of the mojo...or mojo plugin set. Is this the Eclipse profiling stuff? This integration would be pretty tight. Perhaps as you suggest a bit overkill for a first round. What I'm looking to do is to work with a couple guys here over the course of the next 1-3 weeks to erect a solid framework. If we can add some of this profiler spice it would be really cool but not a requirement. FYI I'm a big fan of Yourkit but free tools are much more appealing. > - Something similar for OpenLDAP... > - Bind this mojo to the maven lifecycle in each > archetype. > - Now anyone can create a quick maven DS performance > testing project, make a few configuration file changes > if that's what's being tested, add unit tests if > necessary, and go! Sounds really fun to do: it's a nice little project and the utility will be high. I like the idea of having different mojos for each LDAP server that needs to be setup and leaving setup tasks to the mojo/archetype pair. > - The mojo should also produce put the reports in an > output directory somewhere within the site directory, > and this directory should be the same for all DSs. This will be nice to have ... an easily published set of results. > I'll be glad to throw together a sample archetype for > ApacheDS if this sounds interesting. I'm interested in the idea. I'd love to take a look at it. My Maven 2 archetype skills are non-existent tho. I need to brush them up. > Coding the mojo should also be fairly straightforward. Yeah I agree plugins are a cake walk in m2. SOME NOTES ========== o [Observation] As Quanah suggested in this thread, we need more than one client to really stress test an LDAP server for total throughput under high concurrency. This however is one kind of test we will perform: there are also capacity tests as well. We can still use the recommended tools (SLAMD or another), the partial goal of this effort is to reduce the setup, and running of LDAP servers to be tested in various ways. We want to automate the setup process with an easy way to create/setup a harness for various LDAP servers with similar testing conditions. o Why Maven? It is important to figure out why we (me too btw) want to use Maven. OK PLEASE DON'T FLAME ME ON THIS GUYS :D. First off it's our build tool of choice for this project. Secondly it can run test cases, and fork them if need be with new JVM settings. It's got a nice plugin architecture and has these things for creating project templates called archetypes. BTW Ole, you caught me by surprise when you suggested using archetypes and plugins (mojos) for this effort. The idea gives me a good feeling even though I don't know why yet ... there must be many good advantages to this. Alex > > - Write a maven mojo that > > --- Alex Karasulu <[EMAIL PROTECTED]> wrote: > > Need Benchmarking/Profiling Perf Suite (BPPS) for > ApacheDS > >> ========================================================== > I just sat down in front of my machine and wanted to > run some > performance tests on the server. These questions > came up: > > o What tests do I run? > o What LDIF should I import first before running > those tests? > o Which version of the server should I run these > tests against? > o Can I run these tests against other LDAP servers? > > I did the first thing anyone would do. I tapped > Emmanuel on the > shoulder to ask him for his materials for his AC EU > presentation. I did > not want to repeat the work that he had already > done. > > Please Emmanuel take no offense but I found the > setup and repeated work > to be a bit of a hassle. I'm sure you were bothered > by doing things > manually yourself. Plus I wanted to profile these > tests too inside > Eclipse using Yourkit. Anyway I came to a final > conclusion: > > *Conclusion*: We need some repeatable > benchmarking/profiling perfromance > test suite for ApacheDS that can be run easily. > > > Requirements for BPPS > ===================== > > Here's what I started asking myself for internally. > Please add to this > list if you can think of other requirements. > > (1a) Need repeatable performance tests with setup > prep and tear down > (1b) Tests should be able to load an initial data > set (LDIF) into server > (2) I should be able to use Maven or ant to kick off > these tests > (3) Tests should produce some kind of report > (4) Tests should easily be pointed to benchmark > other servers > (5) Make it easy to create a new performance test. > (6) I want a summary of the conditions in the test > report which include > the setup parameters for: > o operations performed > o capacity, > o concurrency, > o hardware. > o operating system > > > Existing work and potential approaches > ====================================== > > I figured using JUnit was the best way to test > ApacheDS or anyother > server. Plus I could setUp and tearDown test cases. > The only thing I > needed to do was make a base test case or two for > the various apacheds > configurations (embedded testing verses full > networked testing). > > The first base test case, for embedded testing, was > setup here: > > >> http://svn.apache.org/viewvc/directory/trunks/apacheds/core-unit/src/main/java/org/apache/directory/server/core/unit/AbstractPerformanceTest.java?revision=414035&view=markup > Yeah it's weak and I'll try to add to it. What I > would like to do is > invite people to work with me on setting up this > benchmarking/profiling/perf testing framework. > > Comments? Thoughts? > > Thanks, > Alex >> > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEpDJiPVto+tI7JJARApBmAJ9lJgR405jCoOWcNNHvh2oQh5+IoQCguEqR AYSYMMpLRTwaFl2pLZQ8e4E= =r8ed -----END PGP SIGNATURE-----
