in addition to what Grant said, even if your documents are similar, what about queries?
For example, if only a few trec queries contain proper names, acronyms, abbreviations, or whatever, but your users frequently input things like this, it won't be representative. i will disagree with him on a few things though, I would rather have less queries (25 or so), but more judgements, definitely a lot more than 10. Maybe your users only care about the top-10 results but its crucial to judge some lower-ranking docs too, especially if you have recall problems... On Thu, Jan 28, 2010 at 9:34 AM, Grant Ingersoll <gsing...@apache.org>wrote: > > On Jan 27, 2010, at 1:36 PM, Ivan Provalov wrote: > > > Robert, Grant: > > > > Thank you for your replies. > > > > Our goal is to fine-tune our existing system to perform better on > relevance. > > What kind of documents do you have? Are they very similar to the TREC docs > (i.e. news articles)? There can be a fairly wide difference in performance > between real docs and TREC docs, especially given real queries. Doing well > at TREC does not necessarily equate to doing well in your own system. You > might be better off just doing something like taking the top 50 queries from > your logs plus some random ones from the tail and judging the top 10. See > http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Debugging-Relevance-Issues-Search > > > -------------------------- > Grant Ingersoll > http://www.lucidimagination.com/ > > Search the Lucene ecosystem using Solr/Lucene: > http://www.lucidimagination.com/search > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > -- Robert Muir rcm...@gmail.com
