[
https://issues.apache.org/jira/browse/MAHOUT-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169310#comment-13169310
]
Anatoliy Kats commented on MAHOUT-906:
--------------------------------------
I guess it's a hybrid of some sort between estimation and your implementation
of the IR tests. You're right, I used the term test data to what you call
relevant data, and I called the rest training data. The estimation test does
not work for me because at present I am working with boolean preferences. So,
taking into account what you just said, I propose a PrefSplitIRStatsEvaluator
that does the following:
1. Split the data into the training and testing set, with the splitting logic
factorized into a separate class. This class should probably ensure that only
relevant data goes into test set. I have very little experience with
non-boolean data, so I'll follow your directions about where to put the
relevance check.
2. Build a model on the training data
3. Generate the same number of preferences for each user as in the testing data
4. Compute IR statistics on the intersection of actual and predicted
preferences.
The temporal evaluator I need will then loop over calls this class and
aggregate statistics over it. I will certainly implement it for my project,
and I do not mind contributing it to Mahout.
> Allow collaborative filtering evaluators to use custom logic in splitting
> data set
> ----------------------------------------------------------------------------------
>
> Key: MAHOUT-906
> URL: https://issues.apache.org/jira/browse/MAHOUT-906
> Project: Mahout
> Issue Type: Improvement
> Components: Collaborative Filtering
> Affects Versions: 0.5
> Reporter: Anatoliy Kats
> Priority: Minor
> Labels: features
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I want to start a discussion about factoring out the logic used in splitting
> the data set into training and testing. Here is how things stand: There are
> two independent evaluator based classes:
> AbstractDifferenceRecommenderEvaluator, splits all the preferences randomly
> into a training and testing set. GenericRecommenderIRStatsEvaluator takes
> one user at a time, removes their top AT preferences, and counts how many of
> them the system recommends back.
> I have two use cases that both deal with temporal dynamics. In one case,
> there may be expired items that can be used for building a training model,
> but not a test model. In the other, I may want to simulate the behavior of a
> real system by building a preference matrix on days 1-k, and testing on the
> ratings the user generated on the day k+1. In this case, it's not items, but
> preferences(user, item, rating triplets) which may belong only to the
> training set. Before we discuss appropriate design, are there any other use
> cases we need to keep in mind?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
