Great points, Robert! I agree, we have a lot of fine tuning ahead of us.
I think we probably have achieved the baseline with our MAP of 0.14. We should move on to stage two and apply some of the suggestions to improve the overall scores. These are just the first steps. Both you and Grant brought up good points on the users' information needs and the precision/recall tuning of the system. Thanks, Ivan --- On Thu, 1/28/10, Robert Muir <rcm...@gmail.com> wrote: > From: Robert Muir <rcm...@gmail.com> > Subject: Re: Average Precision - TREC-3 > To: java-user@lucene.apache.org > Date: Thursday, January 28, 2010, 11:44 AM > right, but the problem is when > something is currently ranked as doc 20 but > should be in the top 1, 5, or 10, and you aren't seeing > it. > > so I think if you are judging top-N docs from an existing > system, you should > look a little farther ahead than the top-N you care about. > I think you should also index your data a few different > ways and judge those > top-N too, for example, use n-gram tokenizer. > > It doesn't have to be crazy like a formal trec-like pooling > process, but I > think you need to introduce enough variation that you have > judgements for > docs that should be ranked higher than they currently are. > > > > Perfectly reasonable as well. I've seen some > people who only care about > > p...@5 and even p...@1 and others who do much more. > The important thing is to > > think about what makes sense for your application and > users. Much of this > > can be found through basic log analysis (assuming an > existing system) or > > some reasoning about use cases (new system) and users > (how sophisticated > > they are, etc.) > > > > -Grant > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > -- > Robert Muir > rcm...@gmail.com > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
