Hmm, I think you guys are overthinking the details on this; let's step back a level.
When you're running tests, you have these tasks: * create a blank-slate wiki to run tests in * populate the empty wiki with known data * run tests on this wiki in a known state * clean up in some way The parserTests system is designed with some particular constraints: * run within the local MediaWiki test instance a developer already has, without altering its contents * run all tests in a single command-line program * expose no in-test data to the outside world It can do this very easily using temporary tables and a temporary directory because it only needs to work within that one single test process; once it's done, it's done and all the temporary data can be discarded. Nothing needs to be kept across processes or exposed to clients. For Selenium tests, you have a very different set of constraints: * test data must be exposed to web clients over a web server * test data must be retained across multiple requests The simplest way to accomplish this is to have a dedicated, web-exposed test wiki instance. A test run would go like this: * (re)initialize test wiki into known state * run series of tests I'd recommend not trying to use the parserTest harness/initialization for this; it'll be a *lot* simpler to just script creation of a fresh wiki. (drop database, create database, slurp tables, run update.php) Some non-destructive tests can always be run on any existing instance -- and probably should be! -- and some 'active' tests will be freely runnable on existing instances that are used for development and testing, but if you want to work with a blank slate wiki exposed to web clients, keep things simple and just make a dedicated instance. -- brion On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett <dness...@yahoo.com> wrote: > On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: > > > Hi, > > > > here are my thoughts about phpunit and selenium testing. > > > >> The wiki under test is set up with a master database consisting of a > >> single objectcache table. The entries of this table specify a test run > >> identifier as primary key and temporary resource identifiers as > >> dependent fields. > > If I understand this correctly, this would not allow to test any wikis > > that are running on live sites, e.g. intranet wikis. While I agree that > > regression testing on live sites is not a good idea, I kind of like the > > notion that after setting up a wiki with all the extensions I like to > > have, I could do some sort of "everything up and running"-test. With the > > concept of using separate testing databases and resources, this would be > > possible without interference with the actual data and could even be > > done at intervals during, say, maintenance periods. > > The problem with testing live sites is tests may alter wiki data > (consequently, test run reproducibility becomes a problem). If all of the > tests are read-only, then that isn't a problem, but it means developing a > whole set of tests that conform to that constraint. > > Nevertheless, it wouldn't be hard to design the switching mechanism to > allow the testing of live sites. There could be an option in test setup > and cleanup that effectively says "don't switch-in/clean-up temporary > resources." Personally, I think use of this option is dangerous, but it > wouldn't be hard to implement. > > >> Setup of a test run requires the creation of the test run temporary > >> resources and a entry in the objectcache table. > > Are there already mechanisms for this? I haven't done too much work with > > the objectcache. This is where memcached data is stored, right? So how > > do I get the data that is needed? This question leads me to another one: > > How do I get the testind database and resources? As I see this, it > > should be part of the testing framework to be able to produce the set of > > data needed from a "normal" MW installation. The whole mechanism would > > actually be something like a backup, so we might look into any existing > > solutions for that. > > We should use the existing ObjectCache class to manage the object cache. > However, if there exists some switching-in code, I doubt it has > corresponding clean-up code. So, we probably need to do some development > even if we use existing mechanisms. > > I think the object cache and memcached are alternative ways of storing > persistent data. (I also am not an expert in this, so I could be wrong). > My understanding is memcached uses the memcached daemon (http:// > memcached.org/), while the object cache uses the underlying database. If > so, then memcached data disappears after a system crash or power outage, > whereas object cache data should survive. > > You are absolutely correct that we need to figure out how to clone a set > of temporary resources (db, images directory, perhaps cache data) and set > them up for use (i.e., so the switch-in logic can copy them for the test > run). There are a number of problems to solve, e.g., 1) how do you > package the cloned resources (e.g., tar file), 2) how do you efficiently > copy them during the switch-in process, 3) security issues, 4) resource > management issues. > > >> When a request is sent to the wiki under test, very early in the > >> request processing (e.g., immediately after LocalSettings is processed) > >> a hook is called with the provided state information as an argument > >> that accesses the objectcache table. The extension function handling > >> the hook switches in the temporary resources and returns. > > Also, this hook might handle the reconfiguration of the wiki, if needed. > > So for example, testing the PagedTiffHandler requires uploading of tiff > > files to be enabled. However, there might be some security risks, since > > it is not directly obvious in the code which settings are changed. So > > the hook should only be called when $wgEnableSelenium = true. In > > addition, we could define a DontTouchThisSettings.php, which is called > > even after the hook and holds some settings that are holy to the admin > > of the wiki :) The question is, though, would this not become somewhat > > too complicated? > > I have done some research on possible hooks. I think the SetupAfterCache > hook is a good candidate. The problem with calling a (psuedo-)hook in > LocalSettings is there is some setup not completed when LocatSettings is > processed (e.g., the WebRequest object is not yet available. the > ObjectCache class file is not yet included). Most setup is completed > before calling the SetupAfterCache hook. > > It would be possible to do some configuration in this hook, but I think > we need to consider efficiency. Preferably, the creation of the base > resources that are copied during the switch-in will be configured as much > as possible. For example, I think creating the base for PagedTiffHandler > should start with a freshly installed wiki and upload multipage.tiff. The > resulting db could be dumped and the images directory as well as other > resources copied. The result could them be tar'd and uploaded as the base > for the PagedTiffHandler test suite. This would relieve the test set up > code of uploading multipage.tiff on each test run. > > We may even consider pre-installing the base db so that the switch-in > code can use db functions to clone it, rather than creating it from the > base dump for each test run that uses it. The latter could take a > significant amount of time, thereby severely slowing testing. > > >> After the test run completes, the testing application cleans up the > >> test run by requesting the deletion of the temporary resources and the > >> objectcache table entry associated with the test run. > > In some cases, tests will not change any data, e.g. testing dynamic skin > > elements in vector skin. Would it make sense not to tear down the > > testing environment in that case in order to save some time when testing > > repeatedly? I think, there is a conflict between performance and amount > > of data, but who wins? > > There may be ways to make things more efficient by not immediately > deleting the temporary resources. However, eventually we have to delete > them. So, there is a question of where the temporary resources identifier > is stored so it can be used later for a clean-up request. I was assuming > the switch-in request occurs in test suite start-up and the clean-up > request occurs in the test suite finishing code. But, we should consider > alternative implementation strategies. > > > In general, it seems to me that we have some similarity with what is > > called wiki family on mediawiki.org. One could see multiple testing > > environments as a set of multiple wikis that share a common codebase > > [1]. Does anybody have experience with wiki families and the object > > cache rsp. memcached? > > I agree. (In fact, I mentioned this previously). No need to reinvent the > wheel. > > > I am not sure whether we can use the same codebase as parsertests. I'd > > rather think, parser tests are a special case of what we are sketching > > here. On the other hand, I don't think it is a good idea to have two > > separate approaches for very similar tasks in the code. Do you think it > > would be feasible to separate the preparation part from both parser > > tests and selenium tests and build both of them on a common ground? > > > > The parserTests code creates some temporary tables in an existing wiki > database. Currently, these tables are empty and prefixed with a static > identifier. The requirements for parserTests and selenium tests are > significantly different. While we may be able to learn from the > parserTests code, we would have to change the parserTest code > significantly in order to use it. I think it would actually be less work > to start from scratch. Of course, as you mention, there may be code used > to support wiki families that we could use without much modification. > > -- > -- Dan Nessett > > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
