Hi Kathey,
My initial ramblings on this topic start out at the end of August in the
email thread "client/server compatibility testing". There I worried that
over time, the compatibility tests could grow large (taking maybe 5
minutes per combination) and so, if run for all the combinations, would
make derbyall take too long. That's why we're only runing one
combination as part of derbyall. That combination doesn't really track a
compatibility issue, it just tracks regressions which might creep into
the test as other code changes.
I'm certainly in favor of running all the combinations on a nightly or
weekly basis and as a sanity check when cutting release candidates.
I like the idea of wiring some upgrade tests into derbyall. I think it
would tend to enforce the discipline that developers write upgrade logic
and tests at the same time that they checkin code which raises upgrade
problems. Writing upgrade tests is a tricky problem. Technically, an
upgrade test case consists of two pieces: 1) the Setup code which is
compiled and run against the previous release and 2) the Verification
code which is compiled and run against the current release. If you want
to tackle upgrade testing by checking in jar files, you'll have to add
some tricky build logic which builds Setup code against the previous
release and Verification code against the current release.
Just for comparison, here's the equally tricky solution which we adopted
for Cloudscape back in the old days: 1) The Setup code was checked into
the previous release branch, 2) the test developer ran a script to
populate a previous rev database with all the Setup cases then tar up
the resulting database, 3) that tarball was checked into the current
release, 4) the derbyall piece of the upgrade tests then grabbed that
tarball, upgraded it, and ran the Verification code against it.
Cheers,
-Rick
Kathey Marsden wrote:
I noticed that DERBY-516 was closed with the addition of the new test
jdbcapi\CompatibilityTest.java. which adds a nice framework for some
basic compatibility testing.
By default it seems that this test just runs with 10.2 client and 10.2
server so doesn't give us any additional coverage in derbyAll for
compatibility testing. I was wondering if there were plans to check the
derby release jars into the trunk so that the compatibility testing
would be performed as part of the normal testing procedures? These jars
could also be used in DERBY-514 to enable upgrade testing.
Thanks
Kathey
