On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: > On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett <dness...@yahoo.com> > wrote: > >> Some have mentioned the possibility of using the wiki family logic to >> help achieve these objectives. Do you have any thoughts on this? If you >> think it is a good idea, how do we find out more about it? >> >> > I'd just treat it same as any other wiki. Whether you're running one or > multiple wiki instances out of one copy of the code base, it just > doesn't make a difference here. > > -- brion
Oh. Well, perhaps we don't agree exactly 100%. As you suggest, let's step back a bit. Once we get the selenium framework working I assume it will be used for a regression test. This will comprise a set of individual tests. Generally, these tests will write into the wiki db (some may not, but many will). To ensure test reproducibility, the state of the wiki should be the same each time one of these individual tests runs. But, there is a problem. With parserTests, each individual test runs serially. That is fine for parserTests, since (I just ran this on my machine) while there are 610 individual tests, each takes about .08 seconds to run (on average). So, on my machine the whole parserTest takes about 48 seconds. Selenium tests are far more heavy-weight. A rough ball-park figure is each takes about 10 seconds to run (this does not include the time it would take to setup and tear down a "clean wiki"). So, a selenium-based regression test comprising 180 individual tests would take around 30 minutes. Not too bad. But, things are a bit more complicated. Each individual test runs multiple times, once for every browser/OS combination chosen for the regression test. For example, right now there are 13 configured browser/ OS combinations on the WMF Selenium Grid (see http:// grid.tesla.usability.wikimedia.org/console). So even if you only test 4 of these browser/OS configurations, the regression test (if individual tests run serially) would take 2 hours. If you test 8 of them, it would take 4 hours. This is starting to get onerous. If an individual developer wishes to ensure his modifications don't break things before committing his changes, then waiting 4 hours for a regression test to complete is a pretty heavy penalty. Generally, very few will pay the price. So, running the individual tests of a selenium-based regression test serially is not very attractive. This means you need to achieve some concurrency in the regression test. Since individual tests may interfere with each other, you need a way to protect them from each other. This is what the switching functionality is for. You switch in a base set of temporary resources for each test (or perhaps more likely for a particular test suite comprised of mulitple individual tests) consisting of a db, images directory, etc. This allows tests to run without interfering with each other. -- -- Dan Nessett _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
