Hi all,
As we have all known for quite some time, running the full set of geoserver
unit and integration tests takes quite some time. You also may have noticed
that post security changes the tests take even longer on trunk. I have been
spending spare cycles recently looking into how to remedy this. Here are
some thoughts/findings.
1. "Migration" is expensive
Our integration test setup is still using the pre 2.0 data directory
structure. There are some hacks in the geoserver loading that prevent the
conversion to the 2.x style that most of the tests use. Tests that want use
the new setup have the option to set a flag to do so. In this case the old
structure is still written out but once the application context is started
the conversion takes place.
Post security changes, an additional migration takes place that sets up all
the new security services. This is actually quite expensive and doing it
during every test setup / tear down (and by every i mean every "one time"
setup/teardown) burns through a lot of time.
2. Not enough utilization of one time setup
Even without the price paid by the security migration tests still take a
long time to run. The worst offenders appear to be the the wfs and
restconfig modules and the reason being many of the test cases in that
module don't use one time setup because they actually modify data and
configuration. And in order to ensure a fresh copy of data/configuration on
each test case it forgoes one time setup and does the setup every time.
So, how do we go about fixing this.
(1) Keeping the same pattern as the existing test setup, we could simply
serialize out the new structure rather than the old structure. Should be
relatively straight forward and can probably use the facilities in
XStreamPersister to do so. An alternative would be to have a pre-canned
data directory and copy/unpack it when we have to do test setup.
(2) This is the interesting/tricky one. If tests that actually modify data
are going to utilize a one-time setup then what we need is way for a test
case to reload or refresh single layers at a time, returning both
configuration and data to original state. Or in cases where new layers are
added, remove them. Or if existing layers deleted, restore them. All of the
above I believe should really just amount to some utility classes on the
base test classes. However it does add a burden on the test writer since
they now have to worry about initialization/cleanup of state. Also there is
the issue of test order. The order of tests run in an IDE is not always the
same as running from maven and since state is maintained across test
methods i can see this becoming a mess in which failures occur with maven
but not from the ide.
Anyways, as an exercise I decided to locally port all the restconfig tests
to a one-time setup. As expected the results are significant. Currently on
trunk the full restconfig test run takes approximately 5 minutes on my
local machine. Utilizing the one time setup this drops down to about 1
minute 20 seconds.
So can we find a middle ground between using a one-time setup and making
test writers not have to worry too much about state between test methods?
One of the ideas i have is to use annotations to help with this. With
Junit4 we essentially have the ability to define our own custom
annotations, called "rules". So we could do something like this:
@Test
@PostReload
public void testFoo() {
...
}
The "PostReload" annotation would essentially tell the framework that the
test modifies configuration/data and after the test is run the
configuration must be reloaded. This would allow tests mixing test methods
that are read only with ones that modify configuration only pay the price
when things are changed. Unfortunately this could still result in a lot of
configuration reloads for a single test class, like in the case for
restconfig.
Pushing the idea a bit further, i had the idea of being more granular about
it. Doing something like this:
@Test
@PostReload(store=sf)
@PostRemove(layer=sf:newLayer)
public void testFoo() {
...
}
So instead of reloading the entire setup only a single store/layer/etc...
would have to be reloaded, or deleted, etc...
Anyways, just a rough idea at this point. Adopting helper annotations like
this would imply an upgrade of Junit to version 4. Which some tests are
actually already using afaik. There is also TestNG, which from a
customization point of view is much more flexible than Junit 4. Downside is
it is really a completely different framework, the upgrade would be much
less seamless and it has less built in support from our current maven and
eclipse tooling.
So some food for thought at this point. Interested in hearing peoples
thoughts on this.
-Justin
--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Geoserver-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
