So,
> Robert:
> * Cyrus work:
> - some of the Xapian stuff would be good to test with older index db formats.
> - It would be good to run tests with two different binaries, one to create
> old-style data and the other one to see that things still work!
> - this would be a good general thing, both for replication and for our own
> testing!
> - The Xapian issues in the last little while would have been found if we had
> legacy version testing.
>
I did a bunch of work on this ages ago, but I forget about it a lot, and I'm
not sure anyone but me really knows it's there because we all treat our
Cassandane config as set-and-forget.
Cassandane _does_, _sort of_, have support for multiple-version testing. If you
have a look in your cassandane.ini (or cassandane.ini.example), you will see
that there are separate sections for [cyrus default], [cyrus replica] and
[cyrus murder].
When Cassandane spins up multiple Cyrus instances for the
replication/murder/etc tests, it'll usually use the [cyrus default] cyrus
installation, but if there are configured [cyrus replica] and/or [cyrus murder]
installations, it will use those to start up any replica or murder instances
needed by the tests.
So, to do multiple-version testing, you install your desired versions of Cyrus,
tell Cassandane which install to use for the default instance and which to use
for replica/murder instances, then run testrunner.pl normally. The tests
themselves have no awareness that there are multiple versions being run, they
just know they have a main instance and a replica (or murder) instance. (The
gory details are in Cassandane/Instance.pm iirc.)
What it doesn't have support for, is for an individual test to manage the
multiple versions (you are either running multiple versions for all tests, or
none), or for a test to test any more than one pair of versions, or for a
single instance to run two different versions of some Cyrus component. It needs
you to manage the Cyrus installs, and tell it which to use for what purpose,
and then it delivers those instances when a test asks for a replica or murder
instance.
Any tests you write specifically to verify cross-version behaviour will, in a
normal situation (i.e. with [cyrus default] only), not be testing cross-version
behaviour because both instances will be from the same install. A test _can_
interrogate which Cyrus version is being used for each instance by calling
Cassandane::Instance->get_version($instance->{installation}) (installation will
be one of "default", "replica", "murder"... i.e. the foo in the [cyrus foo]
section name from your config) if you need to assert specific behaviours
between specific versions. The :min_version and :max_version magics only look
at the default instance, fyi.
I can't remember what the details of the Xapian issue were, but I think to make
a test for something like this, you'd make a test that sets up the world on the
main instance, then replicates it and runs squatter on the replica, then
asserts that the replica's state is what we expect. If you ran this with "the
older version" as your default cyrus and "the newer version" as your replica
cyrus, then you'd be able to see whether old version -> new version is sane
(with the caveat that the test inherently needs to replicate it, so if
replicating it would mask the problem, then you can't test for it like this...)
> ellie:
> * not sure if Cassandane is the right vehicle for general multiple version
> support
>
The Test::Unit CPAN suite it fundamentally depends on is incomplete and
abandonware, so the whole thing is already fairly precarious. I'm worried that
a substantial refactor will make it unreliable, and if it's unreliable, it's
useless.
Cheers,
ellie
