I claim innocence - I think it was Andrea and Jim. At least it was before my time :-)
I think that the "only go to CITE tests during the release cycle" is the problem - we want help with the release cycle and CITE tests being hidden are standing in the way. As for the other point; we could support the person running the cite tests on IRC / Skype / Email as they make the release and find trouble. Jody On 15/05/2010, at 1:59 AM, Rob Atkinson wrote: > Andrea and Jody worked on some content validation services against WFS > - this is OS - "duckhawk" I think they called it. > > I wonder how far that is away from being able to execute the CITE tests? > > As you pointed out, the good thing about CITE is the tests are > separated and formalised, and not hidden in code. > > Maybe there is a middle ground - run automated tests on build using > duckhawk and only go to formal CITE on the release cycle - because its > going to be hard to get our automated tests confirmed as valid CITE > compliance tests (I guess) in the short term. > > Rob > > On Sat, May 15, 2010 at 1:16 AM, Justin Deoliveira <jdeol...@opengeo.org> > wrote: >> I took a run at doing this a while back with some simple patches to the >> engine sources and met with limited success just running into too many >> issues with random failures. >> >> Quite honestly if I had to look at solving this problem it might be to >> come up with a new and much simpler engine not built on xslt. The way >> the tests are structured makes it easy to pull out the relevant information: >> >> (a) the xml for the requests the engine makes to teh server for a given >> test case >> (b) the assertions the engine makes on the server responses >> >> Some scripts could pull out all the information into a text file or >> something. That coupled with a light java testing framework with some >> utility methods for making this style of assertions could go a long way. >> >> Anyways, crazy idea and probably a pipe dream i know... but it's one >> that has stemmed from hours of debugging xslt when something in a test >> fails. And if it were just simple xslt that would be fine. But its xslt >> that generated xslt that generates xml which makes my head hurt. >> >> 2c. >> >> On 10-05-14 1:31 AM, Andrea Aime wrote: >>> Hi, >>> looking someone struggingling for the n-th time trying to >>> make a release makes me really wonder about the viability >>> of the current release process. >>> >>> In short, imho, the process as-is can be carried on only by a core >>> developer, for the simple reason that running cite tests can >>> uncover issues, and those in turn can be fixed only by >>> someone that knows GS/GT well. >>> >>> And oh, provided you manage to put togheter the necessary black >>> magic to make the cite tests to work. >>> I consider making GS releases a sort of art (and a test of anyone >>> patience). >>> >>> Having CITE tests out of the release process would be a great >>> step in simplifying the process. >>> To do that we have to either give up the confirmation that we're >>> still CITE compliant, or we have to consider CITE as a daily >>> activity, something that runs automatically just as our normal >>> build. >>> >>> Do you think it's possible to fully automate the process? >>> The bits needed are: >>> a) setting up databases: check, it's something that can be done, >>> give the build process a configuration of where the db is >>> and a script that can clean up and rebuild the db and >>> we're in business >>> b) starting up and shutting down GeoServer: check, the Start >>> utility class could be used as a starting point for something >>> that starts a GeoServer with a specific configuration, >>> and eventually shuts it down >>> c) starting up the CITE tests with a specific configuration: >>> uh, dunno... maybe? This is where I see various issues. >>> >>> Issues with teamengine: >>> - to run as part of a maven build we'd need to package up >>> and put on repositories the engine and the test suite. >>> This is redistribution. Is that something we can do? >>> If we cannot, I guess we can still resort to some form >>> of private sharing, marking those jars as "provided" >>> and giving to a selected number of people the files along >>> with the instructions to install them in their maven >>> repositories. >>> - the teamengine already has command line utilities to start >>> a specific test suite, but I see no way to specify the >>> parameters, which are usually asked for in an interactive window. >>> I see no easy way to state the test parameters manually. >>> This looks like a roadblock >>> - Once the suite is run we need to collect the results and >>> declare victory or failure. >>> It's something that can be done by looking into the >>> logs, and seems a bit painful, but doable >>> >>> If we can go past the above issues I believe we could >>> roll a set of maven modules, each designed to have >>> a specific set of dependencies (think about the need >>> to exclude WCS 1.1 in WCS 1.0 tests), each performing >>> a custom setup (db, files, whatever) and running >>> a specific test suite. >>> Basically a module to be built for each of the >>> CITE tests that we want to run. >>> >>> And then imagine Hudson running that every single night. >>> Imagine also that you get a report of failures, and you >>> can run one of the above builds specifying the exact >>> test you want to re-run for debug purposes. >>> >>> May not seem much, but to me, compared to what we have >>> today, feels like Nirvana. >>> >>> Cheers >>> Andrea >>> >>> >>> >> >> >> -- >> Justin Deoliveira >> OpenGeo - http://opengeo.org >> Enterprise support for open source geospatial. >> >> ------------------------------------------------------------------------------ >> >> _______________________________________________ >> Geoserver-devel mailing list >> Geoserver-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/geoserver-devel >> > > ------------------------------------------------------------------------------ > > _______________________________________________ > Geoserver-devel mailing list > Geoserver-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-devel ------------------------------------------------------------------------------ _______________________________________________ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
