Re: Decaying score for old preferences when using the .refresh()
That would be great: Specifically if that is some kind of real usage data, and the results are evaluated against a -without decay- baseline, via A/B tests measuring the increase in conversion. Best Gokhan On Wed, Nov 20, 2013 at 2:28 PM, Cassio Melo melo.cas...@gmail.com wrote: Hi guys, thanks for sharing your experiences on this subject, really appreciated. To summarize the discussion: - The decay of old preference values might loose important historical data in cases where the user has no recent activity (Gokhan) - When using decay (or truncate preferences), the precision of rating prediction may be lower (Pat, Gokhan, Ted) but it might increase conversion rates (Gokhan, Pat) since it reflects recent user intent. - Tweaking the score estimation may be a better approach (Gokhan) I'm doing some experiments with e-commerce data, I'll post the results later. Best regards, Cassio On Fri, Nov 8, 2013 at 5:08 PM, Pat Ferrel pat.fer...@gmail.com wrote: I think the intuition here is, when making an item neighborhood base recommendation, to penalize the contribution of the items that the user has rated a long time ago. I didn't test this in a production recommender system, but I believe this might result in recommendation lists with better conversion rates in certain use cases. It’s only one data point but it was a real ecom recommender with real user data. We did not come to the conclusion above, though there is some truth in it. There are two phenomena at play, similarity of users and items, and recent user intent. Similarity of users decays very slowly if at all. The fact that you and I bought an iPhone 1 makes us similar even though the iPhone 1 is no longer for sale. However you don’t really want to rely on user activity that old to judge recent shopping intent. Mahout conflates these unfortunately. Back to the canonical R = [B’B]H; [B’B] is actually calculated using some similarity metric like log-likihood and RowSimilarityJob. B = preference matrix; user = row, item = column, value = strength perhaps 1 for a purchase. H = user history of preferences in columns, rows = items If you did nothing to decay preferences B’=H If you truncate to use only recent preferences in H then B’ != H Out of the box Mahout requires B’=H, and we got significantly lower precision scores by decaying BOTH B and H. Our conclusion was that this was not really a good idea given our data. If you truncate user preferences to some number of the most recent in H you probably get a lower precision score (as Ted mentions) but our intuition was that the recommendations reflect the most recent user intent. Unfortunately we haven’t A/B tested this conclusion but the candidate for best recommender was using most recent prefs in H and all prefs in B. On Nov 7, 2013, at 11:36 PM, Gokhan Capan gkhn...@gmail.com wrote: On Fri, Nov 8, 2013 at 6:24 AM, Ted Dunning ted.dunn...@gmail.com wrote: On Thu, Nov 7, 2013 at 12:50 AM, Gokhan Capan gkhn...@gmail.com wrote: This particular approach is discussed, and proven to increase the accuracy in Collaborative filtering with Temporal Dynamics by Yehuda Koren. The decay function is parameterized per user, keeping track of how consistent the user behavior is. Note that user-level temporal dynamics does not actually improve the accuracy of ranking. It improves the accuracy of ratings. Yes, the accuracy of rating prediction. Since recommendation quality is primarily a precision@20 sort of activity, improving ratings does no good at all. Item-level temporal dynamics is a different beast. I think the intuition here is, when making an item neighborhood base recommendation, to penalize the contribution of the items that the user has rated a long time ago. I didn't test this in a production recommender system, but I believe this might result in recommendation lists with better conversion rates in certain use cases. Best
Re: Decaying score for old preferences when using the .refresh()
Hi guys, thanks for sharing your experiences on this subject, really appreciated. To summarize the discussion: - The decay of old preference values might loose important historical data in cases where the user has no recent activity (Gokhan) - When using decay (or truncate preferences), the precision of rating prediction may be lower (Pat, Gokhan, Ted) but it might increase conversion rates (Gokhan, Pat) since it reflects recent user intent. - Tweaking the score estimation may be a better approach (Gokhan) I'm doing some experiments with e-commerce data, I'll post the results later. Best regards, Cassio On Fri, Nov 8, 2013 at 5:08 PM, Pat Ferrel pat.fer...@gmail.com wrote: I think the intuition here is, when making an item neighborhood base recommendation, to penalize the contribution of the items that the user has rated a long time ago. I didn't test this in a production recommender system, but I believe this might result in recommendation lists with better conversion rates in certain use cases. It’s only one data point but it was a real ecom recommender with real user data. We did not come to the conclusion above, though there is some truth in it. There are two phenomena at play, similarity of users and items, and recent user intent. Similarity of users decays very slowly if at all. The fact that you and I bought an iPhone 1 makes us similar even though the iPhone 1 is no longer for sale. However you don’t really want to rely on user activity that old to judge recent shopping intent. Mahout conflates these unfortunately. Back to the canonical R = [B’B]H; [B’B] is actually calculated using some similarity metric like log-likihood and RowSimilarityJob. B = preference matrix; user = row, item = column, value = strength perhaps 1 for a purchase. H = user history of preferences in columns, rows = items If you did nothing to decay preferences B’=H If you truncate to use only recent preferences in H then B’ != H Out of the box Mahout requires B’=H, and we got significantly lower precision scores by decaying BOTH B and H. Our conclusion was that this was not really a good idea given our data. If you truncate user preferences to some number of the most recent in H you probably get a lower precision score (as Ted mentions) but our intuition was that the recommendations reflect the most recent user intent. Unfortunately we haven’t A/B tested this conclusion but the candidate for best recommender was using most recent prefs in H and all prefs in B. On Nov 7, 2013, at 11:36 PM, Gokhan Capan gkhn...@gmail.com wrote: On Fri, Nov 8, 2013 at 6:24 AM, Ted Dunning ted.dunn...@gmail.com wrote: On Thu, Nov 7, 2013 at 12:50 AM, Gokhan Capan gkhn...@gmail.com wrote: This particular approach is discussed, and proven to increase the accuracy in Collaborative filtering with Temporal Dynamics by Yehuda Koren. The decay function is parameterized per user, keeping track of how consistent the user behavior is. Note that user-level temporal dynamics does not actually improve the accuracy of ranking. It improves the accuracy of ratings. Yes, the accuracy of rating prediction. Since recommendation quality is primarily a precision@20 sort of activity, improving ratings does no good at all. Item-level temporal dynamics is a different beast. I think the intuition here is, when making an item neighborhood base recommendation, to penalize the contribution of the items that the user has rated a long time ago. I didn't test this in a production recommender system, but I believe this might result in recommendation lists with better conversion rates in certain use cases. Best
Re: Decaying score for old preferences when using the .refresh()
I think the intuition here is, when making an item neighborhood base recommendation, to penalize the contribution of the items that the user has rated a long time ago. I didn't test this in a production recommender system, but I believe this might result in recommendation lists with better conversion rates in certain use cases. It’s only one data point but it was a real ecom recommender with real user data. We did not come to the conclusion above, though there is some truth in it. There are two phenomena at play, similarity of users and items, and recent user intent. Similarity of users decays very slowly if at all. The fact that you and I bought an iPhone 1 makes us similar even though the iPhone 1 is no longer for sale. However you don’t really want to rely on user activity that old to judge recent shopping intent. Mahout conflates these unfortunately. Back to the canonical R = [B’B]H; [B’B] is actually calculated using some similarity metric like log-likihood and RowSimilarityJob. B = preference matrix; user = row, item = column, value = strength perhaps 1 for a purchase. H = user history of preferences in columns, rows = items If you did nothing to decay preferences B’=H If you truncate to use only recent preferences in H then B’ != H Out of the box Mahout requires B’=H, and we got significantly lower precision scores by decaying BOTH B and H. Our conclusion was that this was not really a good idea given our data. If you truncate user preferences to some number of the most recent in H you probably get a lower precision score (as Ted mentions) but our intuition was that the recommendations reflect the most recent user intent. Unfortunately we haven’t A/B tested this conclusion but the candidate for best recommender was using most recent prefs in H and all prefs in B. On Nov 7, 2013, at 11:36 PM, Gokhan Capan gkhn...@gmail.com wrote: On Fri, Nov 8, 2013 at 6:24 AM, Ted Dunning ted.dunn...@gmail.com wrote: On Thu, Nov 7, 2013 at 12:50 AM, Gokhan Capan gkhn...@gmail.com wrote: This particular approach is discussed, and proven to increase the accuracy in Collaborative filtering with Temporal Dynamics by Yehuda Koren. The decay function is parameterized per user, keeping track of how consistent the user behavior is. Note that user-level temporal dynamics does not actually improve the accuracy of ranking. It improves the accuracy of ratings. Yes, the accuracy of rating prediction. Since recommendation quality is primarily a precision@20 sort of activity, improving ratings does no good at all. Item-level temporal dynamics is a different beast. I think the intuition here is, when making an item neighborhood base recommendation, to penalize the contribution of the items that the user has rated a long time ago. I didn't test this in a production recommender system, but I believe this might result in recommendation lists with better conversion rates in certain use cases. Best
Re: Decaying score for old preferences when using the .refresh()
Cassio, I am not sure if there are direct/indirect ways to to this with existing code. Recall that an item neighborhood based score prediction, in simplest terms, is a weighted average of the active user's ratings on other items, where the weights are item-to-item similarities. Applying a decay function to these item-to-item weights, where the the decay is based on the rating time of the active user on the other items can help to achieve this. One consideration might be for users who do not change their rating behavior much, this decay can mask valuable historical information. This particular approach is discussed, and proven to increase the accuracy in Collaborative filtering with Temporal Dynamics by Yehuda Koren. The decay function is parameterized per user, keeping track of how consistent the user behavior is. If you think it is not necessary to estimate those per user parameters, in Mahout's current neighborhood based recommenders, you might apply that decay to item-to-item similarities at recommendation time. Note that DataModel#getPreferenceTime is the method you require. If you're using a GenericItemBasedRecommender directly, GenericItemBasedRecommender#doEstimatePreference is where your edits would go. The benefit here is not having to update item-to-item similarities, so you can still cache them. Gokhan On Wed, Nov 6, 2013 at 6:32 PM, Cassio Melo melo.cas...@gmail.com wrote: Assuming that most recent ratings or implicit preference data is more important than the older ones, I wonder if there is a way to decrease the importance (score) of old preference entries without having to update all previous preferences. Currently I'm fetching new preferences from time to time and using the .refresh() method to update the data model with the new values. Thanks
Re: Decaying score for old preferences when using the .refresh()
Not sure how you are going to decay in Mahout. Once ingested into Mahout there are no timestamps. So you’ll have to do that before ingesting. Last year we set up an ecom-department store type recommender with data from online user purchase, add-to-cart, and view. The data was actual user behavior in the online store before any recommender was implemented. So it was very clean of external effects. We took varying time slices and measured change in precision of Mahout’s item-base recommendations. We found that the precision always increased with more data up to our max of 1 year. Put another way we took 3 months, 6 months, 9 months, and 12 months produced the best results. All we did was filter items no longer in stock. We did nothing to decay preferences. That said you can still make a good case to limit or decay user preferences used in the queries. The problem is you may not want to have the same limit on data used to build the model. The model data represents user’s taste similarities, which change very slowly. I don’t know of a way to have a short time span user preference query against a long time span model in Mahout, as Gokhan says. If you care to hack Mahout you can use different data in the recommendation pipeline. Mahout uses the user preference matrix to calculate item-item similarities and puts them in a DRM (distributed row matrix) then it uses the user’s preference data taken from the preference matrix as a sort of query agains the item-item DRM. If you use your own truncated user preference vectors (or decayed) as the queries instead of the ones that were used to train the item-item DRM you would do get the result you are trying for without throwing out potentially important training data. By decaying the user preferences you may get a lower precision score, but that is only a crude measure of goodness. The recs for recent user activity will probably result in more sales since they indicate recent user intent. You can measure this later with A/B testing if you want. On Nov 7, 2013, at 12:50 AM, Gokhan Capan gkhn...@gmail.com wrote: Cassio, I am not sure if there are direct/indirect ways to to this with existing code. Recall that an item neighborhood based score prediction, in simplest terms, is a weighted average of the active user's ratings on other items, where the weights are item-to-item similarities. Applying a decay function to these item-to-item weights, where the the decay is based on the rating time of the active user on the other items can help to achieve this. One consideration might be for users who do not change their rating behavior much, this decay can mask valuable historical information. This particular approach is discussed, and proven to increase the accuracy in Collaborative filtering with Temporal Dynamics by Yehuda Koren. The decay function is parameterized per user, keeping track of how consistent the user behavior is. If you think it is not necessary to estimate those per user parameters, in Mahout's current neighborhood based recommenders, you might apply that decay to item-to-item similarities at recommendation time. Note that DataModel#getPreferenceTime is the method you require. If you're using a GenericItemBasedRecommender directly, GenericItemBasedRecommender#doEstimatePreference is where your edits would go. The benefit here is not having to update item-to-item similarities, so you can still cache them. Gokhan On Wed, Nov 6, 2013 at 6:32 PM, Cassio Melo melo.cas...@gmail.com wrote: Assuming that most recent ratings or implicit preference data is more important than the older ones, I wonder if there is a way to decrease the importance (score) of old preference entries without having to update all previous preferences. Currently I'm fetching new preferences from time to time and using the .refresh() method to update the data model with the new values. Thanks
Re: Decaying score for old preferences when using the .refresh()
On Thu, Nov 7, 2013 at 12:50 AM, Gokhan Capan gkhn...@gmail.com wrote: This particular approach is discussed, and proven to increase the accuracy in Collaborative filtering with Temporal Dynamics by Yehuda Koren. The decay function is parameterized per user, keeping track of how consistent the user behavior is. Note that user-level temporal dynamics does not actually improve the accuracy of ranking. It improves the accuracy of ratings. Since recommendation quality is primarily a precision@20 sort of activity, improving ratings does no good at all. Item-level temporal dynamics is a different beast.
Re: Decaying score for old preferences when using the .refresh()
On Fri, Nov 8, 2013 at 6:24 AM, Ted Dunning ted.dunn...@gmail.com wrote: On Thu, Nov 7, 2013 at 12:50 AM, Gokhan Capan gkhn...@gmail.com wrote: This particular approach is discussed, and proven to increase the accuracy in Collaborative filtering with Temporal Dynamics by Yehuda Koren. The decay function is parameterized per user, keeping track of how consistent the user behavior is. Note that user-level temporal dynamics does not actually improve the accuracy of ranking. It improves the accuracy of ratings. Yes, the accuracy of rating prediction. Since recommendation quality is primarily a precision@20 sort of activity, improving ratings does no good at all. Item-level temporal dynamics is a different beast. I think the intuition here is, when making an item neighborhood base recommendation, to penalize the contribution of the items that the user has rated a long time ago. I didn't test this in a production recommender system, but I believe this might result in recommendation lists with better conversion rates in certain use cases. Best
