Ultimately this is dependent on what your metrics for success are. For some places it may be just raw CTR (did my click through rate increase) but for other places it may be a function of money (either it may be gross revenue, profits, # items sold etc). I don't know if there is a generic answer for this question which is leading those to write their own frameworks b/c it's very specific to your needs. A scoring change that leads to an increase in CTR may not necessarily lead to an increase in the metric that makes your business go.
On Tue, Feb 12, 2013 at 10:31 PM, Steffen Elberg Godskesen < steffen.godske...@gmail.com> wrote: > > Hi Roman, > > If you're looking for regression testing then > https://github.com/sul-dlss/rspec-solr might be worth looking at. If > you're not a ruby shop, doing something similar in another language > shouldn't be to hard. > > > The basic idea is that you setup a set of tests like > > "If the query is X, then the document with id Y should be in the first 10 > results" > "If the query is S, then a document with title T should be the first > result" > "If the query is P, then a document with author Q should not be in the > first 10 result" > > and that you run these whenever you tune your scoring formula to ensure > that you haven't introduced unintended effects. New ideas/requirements for > your relevance ranking should always result in writing new tests - that > will probably fail until you tune your scoring formula. This is certainly > no magic bullet, but it will give you some confidence that you didn't make > things worse. And - in my humble opinion - it also gives you the benefit of > discouraging you from tuning your scoring just for fun. To put it bluntly: > if you cannot write up a requirement in form of a test, you probably have > no need to tune your scoring. > > > Regards, > > -- > Steffen > > > > On Tuesday, February 12, 2013 at 23:03 , Roman Chyla wrote: > > > Hi, > > I do realize this is a very broad question, but still I need to ask it. > > Suppose you make a change into the scoring formula. How do you > > test/know/see what impact it had? Any framework out there? > > > > It seems like people are writing their own tools to measure relevancy. > > > > Thanks for any pointers, > > > > roman > > >
