Thanks for the input guys. I've decided to implement some unit tests for now, although we don't have a clean data set to work from (sucks, I know).
We're going to keep track of a set of vital queries, and ensure they don't return 0 results, as we have a pretty decent level of confidence with Solr's text matching. So not ideal, but better than nothing ;o) That should find anything that's gone horribly wrong, while at the same time dealing with our data set changing, and us not having very brittle tests. Much appreciated, Mark On Wed, Apr 6, 2011 at 6:54 PM, Paul Libbrecht <p...@hoplahup.net> wrote: > Mark, > > In one project, with Lucene not Solr, I also use a smallish unit test > sample and apply some queries there. > It is very limited but is automatable. > > I find a better way is to have precision and recall measures of real users > run release after release. > I could never fully apply this yet on a recurring basis sadly. > > My ideal world would be that the search sample is small enough and that > users are able to restrict search to this. > Then users have the possibility of checking correctness of each result > (say, first 10) for each query out of which one can then read results. > Often, users provide comments along, e.g. missing matches. This is packed as > a wiki page. > First samples generally do not use enough of the features, this is adjusted > as a dialogue. > > As a developer I review the test suite run and plan for next adjustments. > The numeric approach allows easy mean precision and mean recall which is > good for reporting. > > My best reference for PR testing and other forms of testing Kavi Mahesh's > Text Retrieval Quality, a primer: > http://www.oracle.com/technetwork/database/enterprise-edition/imt-quality-092464.html > > I would love to hear more of what the users have been doing. > > paul > > > Le 6 avr. 2011 à 08:10, Mark Mandel a écrit : > > > Hey guys, > > > > I'm wondering how people are managing regression testing, in particular > with > > things like text based search. > > > > I.e. if you change how fields are indexed or change boosts in dismax, > > ensuring that doesn't mean that critical queries are showing bad data. > > > > The obvious answer to me was using unit tests. These may be brittle as > some > > index data can change over time, but I couldn't think of a better way. > > > > How is everyone else solving this problem? > > > > Cheers, > > > > Mark > > > > -- > > E: mark.man...@gmail.com > > T: http://www.twitter.com/neurotic > > W: www.compoundtheory.com > > > > cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia > > http://www.cfobjective.com.au > > > > Hands-on ColdFusion ORM Training > > www.ColdFusionOrmTraining.com > > -- E: mark.man...@gmail.com T: http://www.twitter.com/neurotic W: www.compoundtheory.com cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia http://www.cfobjective.com.au Hands-on ColdFusion ORM Training www.ColdFusionOrmTraining.com
