I'm trying to evaluate a few different recommenders based on boolean
preferences. The in action book suggests using an precision/recall
metric, but I'm not sure I understand what that does, and in particular
how it is dividing my data into test/train sets.
What I think I'd like to do is:
1.
In point 1, I don't think I'd say it that way. It's not true that
test/training is divided by user, because every user would either be
100% in the training or 100% in the test data. Instead you hold out
part of the data for each user, or at least, for some subset of users.
Then you can see whether
Since I am primarily an HPC person, probably a naive question from the ML
perspective.
What if, when computing recommendations, we don't exclude what the user
already has,
and then see if the items he has end up being recommended to him (compute
some appropriate metric / ratio)?
Wouldn't that be
Thanks for your help
Yes, I think a time-based division of test v. training probably would
make sense since that will correspond to our actual intended practice.
But before I worry about that I seem to have some more fundamental
problem that is giving me 0 precision and 0 recall all the
It depends on the algorithm I suppose. In some cases, the
already-known items would always be top recommendations and the test
would tell you nothing. Just like in an RMSE test -- if you already
know the right answers your score is always a perfect 0.
But in some cases I agree you could get some
On Fri, Jun 7, 2013 at 4:50 PM, Sean Owen sro...@gmail.com wrote:
It depends on the algorithm I suppose. In some cases, the
already-known items would always be top recommendations and the test
would tell you nothing. Just like in an RMSE test -- if you already
know the right answers your
But why would she want the things she has?
- Original Message -
From: Koobas [mailto:koo...@gmail.com]
Sent: Friday, June 07, 2013 08:06 PM
To: user@mahout.apache.org user@mahout.apache.org
Subject: Re: evaluating recommender with boolean prefs
Since I am primarily an HPC person,
Yes it makes sense in the case of for example ALS.
With or without this idea, the more general point is that this result
is still problematic. It is somewhat useful in comparing in a relative
sense; I'd rather have a recommender that stacks my input values
somewhere near the top than bottom. But
I believe the suggestion is just for purposes of evaluation. You would
not return these items in practice, yes.
Although there are cases where you do want to return known items. For
example, maybe you are modeling user interaction with restaurant
categories. This could be useful, because as soon
