Re: matrix factorization cross validation
I added the drivers for precisionAt(k: Int) driver for the movielens test-cases...Although I am a bit confused on precisionAt(k: Int) code from RankingMetrics.scala... While cross validating, I am really not sure how to set K... if (labSet.nonEmpty) { val n = math.min(pred.length, k) ... } If I make k as a function of pred.length val n = math.min(pred.length, k*pred.length) then I can vary k between 0 and 1 and choose the sweet spot for K on a given dataset but I am not sure if it is a measure that makes sense for recommendation... MAP is something that makes sense as it is average over all test set... On Fri, Oct 31, 2014 at 1:26 AM, Sean Owen wrote: > No, excepting approximate methods like LSH to figure out the > relatively small set of candidates for the users in the partition, and > broadcast or join those. > > On Fri, Oct 31, 2014 at 5:45 AM, Nick Pentreath > wrote: > > Sean, re my point earlier do you know a more efficient way to compute > top k > > for each user, other than to broadcast the item factors? > > > > (I guess one can use the new asymmetric lsh paper perhaps to assist) > > > > — > > Sent from Mailbox > > > > > > On Thu, Oct 30, 2014 at 11:24 PM, Sean Owen wrote: > >> > >> MAP is effectively an average over all k from 1 to min(# > >> recommendations, # items rated) Getting first recommendations right is > >> more important than the last. > >> > >> On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das < > debasish.da...@gmail.com> > >> wrote: > >> > Does it make sense to have a user specific K or K is considered same > >> > over > >> > all users ? > >> > > >> > Intuitively the users who watches more movies should get a higher K > than > >> > the > >> > others... > >> > > > > > >
Re: matrix factorization cross validation
No, excepting approximate methods like LSH to figure out the relatively small set of candidates for the users in the partition, and broadcast or join those. On Fri, Oct 31, 2014 at 5:45 AM, Nick Pentreath wrote: > Sean, re my point earlier do you know a more efficient way to compute top k > for each user, other than to broadcast the item factors? > > (I guess one can use the new asymmetric lsh paper perhaps to assist) > > — > Sent from Mailbox > > > On Thu, Oct 30, 2014 at 11:24 PM, Sean Owen wrote: >> >> MAP is effectively an average over all k from 1 to min(# >> recommendations, # items rated) Getting first recommendations right is >> more important than the last. >> >> On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das >> wrote: >> > Does it make sense to have a user specific K or K is considered same >> > over >> > all users ? >> > >> > Intuitively the users who watches more movies should get a higher K than >> > the >> > others... >> > > > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: matrix factorization cross validation
Sean, re my point earlier do you know a more efficient way to compute top k for each user, other than to broadcast the item factors? (I guess one can use the new asymmetric lsh paper perhaps to assist) — Sent from Mailbox On Thu, Oct 30, 2014 at 11:24 PM, Sean Owen wrote: > MAP is effectively an average over all k from 1 to min(# > recommendations, # items rated) Getting first recommendations right is > more important than the last. > On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das > wrote: >> Does it make sense to have a user specific K or K is considered same over >> all users ? >> >> Intuitively the users who watches more movies should get a higher K than the >> others... >>
Re: matrix factorization cross validation
MAP is effectively an average over all k from 1 to min(# recommendations, # items rated) Getting first recommendations right is more important than the last. On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das wrote: > Does it make sense to have a user specific K or K is considered same over > all users ? > > Intuitively the users who watches more movies should get a higher K than the > others... > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: matrix factorization cross validation
Does it make sense to have a user specific K or K is considered same over all users ? Intuitively the users who watches more movies should get a higher K than the others... On Thu, Oct 30, 2014 at 2:15 PM, Sean Owen wrote: > The pretty standard metric for recommenders is mean average precision, > and RankingMetrics will already do that as-is. I don't know that a > confusion matrix for this binary classification does much. > > > On Thu, Oct 30, 2014 at 9:41 PM, Debasish Das > wrote: > > I am working on it...I will open up a JIRA once I see some results.. > > > > Idea is to come up with a test train set based on users...basically for > > each user, we come up with 80% train data and 20% test data... > > > > Now we pick up a K (each user should have a different K based on the > movies > > he watched so some multiplier) and then we get topK for each user and see > > the confusion matrix for each user... > > > > This data will also go to RankingMetrics I think...one is ground truth > > array and the other is our prediction...I would like to see the raw > > confusions as well.. > > > > These measures are necessary to validate any of the topic modeling > > algorithms as well... > > > > Is there a better place for it other than mllib examples ? > > > > On Thu, Oct 30, 2014 at 8:13 AM, Debasish Das > > wrote: > > > >> I thought topK will save us...for each user we have 1xrank...now our > movie > >> factor is a RDD...we pick topK movie factors based on vector > norm...with K > >> = 50, we will have 50 vectors * num_executors in a RDD...with the user > >> 1xrank we do a distributed dot product using RowMatrix APIs... > >> > >> May be we can't find topK using vector norm on movie factors... > >> > >> On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath < > nick.pentre...@gmail.com> > >> wrote: > >> > >>> Looking at > >>> > https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82 > >>> > >>> For each user in test set, you generate an Array of top K predicted > item > >>> ids (Int or String probably), and an Array of ground truth item ids > (the > >>> known rated or liked items in the test set for that user), and pass > that to > >>> precisionAt(k) to compute MAP@k (Actually this method name is a bit > >>> misleading - it should be meanAveragePrecisionAt where the other method > >>> there is without a cutoff at k. However, both compute MAP). > >>> > >>> The challenge at scale is actually computing all the top Ks for each > >>> user, as it requires broadcasting all the item factors (unless there > is a > >>> smarter way?) > >>> > >>> I wonder if it is possible to extend the DIMSUM idea to computing top K > >>> matrix multiply between the user and item factor matrices, as opposed > to > >>> all-pairs similarity of one matrix? > >>> > >>> On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das < > debasish.da...@gmail.com> > >>> wrote: > >>> > Is there an example of how to use RankingMetrics ? > > Let's take the user, document example...we get user x topic and > document > x > topic matrices as the model... > > Now for each user, we can generate topK document by doing a sort on > (1 x > topic)dot(topic x document) and picking topK... > > Is it possible to validate such a topK finding algorithm using > RankingMetrics ? > > > On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng > wrote: > > > Let's narrow the context from matrix factorization to recommendation > > via ALS. It adds extra complexity if we treat it as a multi-class > > classification problem. ALS only outputs a single value for each > > prediction, which is hard to convert to probability distribution > over > > the 5 rating levels. Treating it as a binary classification problem > or > > a ranking problem does make sense. The RankingMetricc is in master. > > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC > > should be good to add as well. -Xiangrui > > > > > > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das < > debasish.da...@gmail.com> > > wrote: > > > Hi, > > > > > > In the current factorization flow, we cross validate on the test > dataset > > > using the RMSE number but there are some other measures which are > worth > > > looking into. > > > > > > If we consider the problem as a regression problem and the ratings > 1-5 > > are > > > considered as 5 classes, it is possible to generate a confusion > matrix > > > using MultiClassMetrics.scala > > > > > > If the ratings are only 0/1 (like from the spotify demo from spark > > summit) > > > then it is possible to use Binary Classification Metrices to come > up > with > > > the ROC curve... > > > > > > For topK user/products we should also look into prec@k and
Re: matrix factorization cross validation
The pretty standard metric for recommenders is mean average precision, and RankingMetrics will already do that as-is. I don't know that a confusion matrix for this binary classification does much. On Thu, Oct 30, 2014 at 9:41 PM, Debasish Das wrote: > I am working on it...I will open up a JIRA once I see some results.. > > Idea is to come up with a test train set based on users...basically for > each user, we come up with 80% train data and 20% test data... > > Now we pick up a K (each user should have a different K based on the movies > he watched so some multiplier) and then we get topK for each user and see > the confusion matrix for each user... > > This data will also go to RankingMetrics I think...one is ground truth > array and the other is our prediction...I would like to see the raw > confusions as well.. > > These measures are necessary to validate any of the topic modeling > algorithms as well... > > Is there a better place for it other than mllib examples ? > > On Thu, Oct 30, 2014 at 8:13 AM, Debasish Das > wrote: > >> I thought topK will save us...for each user we have 1xrank...now our movie >> factor is a RDD...we pick topK movie factors based on vector norm...with K >> = 50, we will have 50 vectors * num_executors in a RDD...with the user >> 1xrank we do a distributed dot product using RowMatrix APIs... >> >> May be we can't find topK using vector norm on movie factors... >> >> On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath >> wrote: >> >>> Looking at >>> https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82 >>> >>> For each user in test set, you generate an Array of top K predicted item >>> ids (Int or String probably), and an Array of ground truth item ids (the >>> known rated or liked items in the test set for that user), and pass that to >>> precisionAt(k) to compute MAP@k (Actually this method name is a bit >>> misleading - it should be meanAveragePrecisionAt where the other method >>> there is without a cutoff at k. However, both compute MAP). >>> >>> The challenge at scale is actually computing all the top Ks for each >>> user, as it requires broadcasting all the item factors (unless there is a >>> smarter way?) >>> >>> I wonder if it is possible to extend the DIMSUM idea to computing top K >>> matrix multiply between the user and item factor matrices, as opposed to >>> all-pairs similarity of one matrix? >>> >>> On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das >>> wrote: >>> Is there an example of how to use RankingMetrics ? Let's take the user, document example...we get user x topic and document x topic matrices as the model... Now for each user, we can generate topK document by doing a sort on (1 x topic)dot(topic x document) and picking topK... Is it possible to validate such a topK finding algorithm using RankingMetrics ? On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng wrote: > Let's narrow the context from matrix factorization to recommendation > via ALS. It adds extra complexity if we treat it as a multi-class > classification problem. ALS only outputs a single value for each > prediction, which is hard to convert to probability distribution over > the 5 rating levels. Treating it as a binary classification problem or > a ranking problem does make sense. The RankingMetricc is in master. > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC > should be good to add as well. -Xiangrui > > > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das < debasish.da...@gmail.com> > wrote: > > Hi, > > > > In the current factorization flow, we cross validate on the test dataset > > using the RMSE number but there are some other measures which are worth > > looking into. > > > > If we consider the problem as a regression problem and the ratings 1-5 > are > > considered as 5 classes, it is possible to generate a confusion matrix > > using MultiClassMetrics.scala > > > > If the ratings are only 0/1 (like from the spotify demo from spark > summit) > > then it is possible to use Binary Classification Metrices to come up with > > the ROC curve... > > > > For topK user/products we should also look into prec@k and pdcg@k as the > > metric.. > > > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in > > examples.MovielensALS along with RMSE ? > > > > Thanks. > > Deb > >>> >>> >> - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: matrix factorization cross validation
I am working on it...I will open up a JIRA once I see some results.. Idea is to come up with a test train set based on users...basically for each user, we come up with 80% train data and 20% test data... Now we pick up a K (each user should have a different K based on the movies he watched so some multiplier) and then we get topK for each user and see the confusion matrix for each user... This data will also go to RankingMetrics I think...one is ground truth array and the other is our prediction...I would like to see the raw confusions as well.. These measures are necessary to validate any of the topic modeling algorithms as well... Is there a better place for it other than mllib examples ? On Thu, Oct 30, 2014 at 8:13 AM, Debasish Das wrote: > I thought topK will save us...for each user we have 1xrank...now our movie > factor is a RDD...we pick topK movie factors based on vector norm...with K > = 50, we will have 50 vectors * num_executors in a RDD...with the user > 1xrank we do a distributed dot product using RowMatrix APIs... > > May be we can't find topK using vector norm on movie factors... > > On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath > wrote: > >> Looking at >> https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82 >> >> For each user in test set, you generate an Array of top K predicted item >> ids (Int or String probably), and an Array of ground truth item ids (the >> known rated or liked items in the test set for that user), and pass that to >> precisionAt(k) to compute MAP@k (Actually this method name is a bit >> misleading - it should be meanAveragePrecisionAt where the other method >> there is without a cutoff at k. However, both compute MAP). >> >> The challenge at scale is actually computing all the top Ks for each >> user, as it requires broadcasting all the item factors (unless there is a >> smarter way?) >> >> I wonder if it is possible to extend the DIMSUM idea to computing top K >> matrix multiply between the user and item factor matrices, as opposed to >> all-pairs similarity of one matrix? >> >> On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das >> wrote: >> >>> Is there an example of how to use RankingMetrics ? >>> >>> Let's take the user, document example...we get user x topic and document >>> x >>> topic matrices as the model... >>> >>> Now for each user, we can generate topK document by doing a sort on (1 x >>> topic)dot(topic x document) and picking topK... >>> >>> Is it possible to validate such a topK finding algorithm using >>> RankingMetrics ? >>> >>> >>> On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng >>> wrote: >>> >>> > Let's narrow the context from matrix factorization to recommendation >>> > via ALS. It adds extra complexity if we treat it as a multi-class >>> > classification problem. ALS only outputs a single value for each >>> > prediction, which is hard to convert to probability distribution over >>> > the 5 rating levels. Treating it as a binary classification problem or >>> > a ranking problem does make sense. The RankingMetricc is in master. >>> > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC >>> > should be good to add as well. -Xiangrui >>> > >>> > >>> > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das < >>> debasish.da...@gmail.com> >>> > wrote: >>> > > Hi, >>> > > >>> > > In the current factorization flow, we cross validate on the test >>> dataset >>> > > using the RMSE number but there are some other measures which are >>> worth >>> > > looking into. >>> > > >>> > > If we consider the problem as a regression problem and the ratings >>> 1-5 >>> > are >>> > > considered as 5 classes, it is possible to generate a confusion >>> matrix >>> > > using MultiClassMetrics.scala >>> > > >>> > > If the ratings are only 0/1 (like from the spotify demo from spark >>> > summit) >>> > > then it is possible to use Binary Classification Metrices to come up >>> with >>> > > the ROC curve... >>> > > >>> > > For topK user/products we should also look into prec@k and pdcg@k >>> as the >>> > > metric.. >>> > > >>> > > Does it make sense to add the multiclass metric and prec@k, pdcg@k >>> in >>> > > examples.MovielensALS along with RMSE ? >>> > > >>> > > Thanks. >>> > > Deb >>> > >>> >> >> >
Re: matrix factorization cross validation
I thought topK will save us...for each user we have 1xrank...now our movie factor is a RDD...we pick topK movie factors based on vector norm...with K = 50, we will have 50 vectors * num_executors in a RDD...with the user 1xrank we do a distributed dot product using RowMatrix APIs... May be we can't find topK using vector norm on movie factors... On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath wrote: > Looking at > https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82 > > For each user in test set, you generate an Array of top K predicted item > ids (Int or String probably), and an Array of ground truth item ids (the > known rated or liked items in the test set for that user), and pass that to > precisionAt(k) to compute MAP@k (Actually this method name is a bit > misleading - it should be meanAveragePrecisionAt where the other method > there is without a cutoff at k. However, both compute MAP). > > The challenge at scale is actually computing all the top Ks for each user, > as it requires broadcasting all the item factors (unless there is a smarter > way?) > > I wonder if it is possible to extend the DIMSUM idea to computing top K > matrix multiply between the user and item factor matrices, as opposed to > all-pairs similarity of one matrix? > > On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das > wrote: > >> Is there an example of how to use RankingMetrics ? >> >> Let's take the user, document example...we get user x topic and document x >> topic matrices as the model... >> >> Now for each user, we can generate topK document by doing a sort on (1 x >> topic)dot(topic x document) and picking topK... >> >> Is it possible to validate such a topK finding algorithm using >> RankingMetrics ? >> >> >> On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng wrote: >> >> > Let's narrow the context from matrix factorization to recommendation >> > via ALS. It adds extra complexity if we treat it as a multi-class >> > classification problem. ALS only outputs a single value for each >> > prediction, which is hard to convert to probability distribution over >> > the 5 rating levels. Treating it as a binary classification problem or >> > a ranking problem does make sense. The RankingMetricc is in master. >> > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC >> > should be good to add as well. -Xiangrui >> > >> > >> > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das < >> debasish.da...@gmail.com> >> > wrote: >> > > Hi, >> > > >> > > In the current factorization flow, we cross validate on the test >> dataset >> > > using the RMSE number but there are some other measures which are >> worth >> > > looking into. >> > > >> > > If we consider the problem as a regression problem and the ratings 1-5 >> > are >> > > considered as 5 classes, it is possible to generate a confusion matrix >> > > using MultiClassMetrics.scala >> > > >> > > If the ratings are only 0/1 (like from the spotify demo from spark >> > summit) >> > > then it is possible to use Binary Classification Metrices to come up >> with >> > > the ROC curve... >> > > >> > > For topK user/products we should also look into prec@k and pdcg@k as >> the >> > > metric.. >> > > >> > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in >> > > examples.MovielensALS along with RMSE ? >> > > >> > > Thanks. >> > > Deb >> > >> > >
Re: matrix factorization cross validation
Looking at https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82 For each user in test set, you generate an Array of top K predicted item ids (Int or String probably), and an Array of ground truth item ids (the known rated or liked items in the test set for that user), and pass that to precisionAt(k) to compute MAP@k (Actually this method name is a bit misleading - it should be meanAveragePrecisionAt where the other method there is without a cutoff at k. However, both compute MAP). The challenge at scale is actually computing all the top Ks for each user, as it requires broadcasting all the item factors (unless there is a smarter way?) I wonder if it is possible to extend the DIMSUM idea to computing top K matrix multiply between the user and item factor matrices, as opposed to all-pairs similarity of one matrix? On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das wrote: > Is there an example of how to use RankingMetrics ? > > Let's take the user, document example...we get user x topic and document x > topic matrices as the model... > > Now for each user, we can generate topK document by doing a sort on (1 x > topic)dot(topic x document) and picking topK... > > Is it possible to validate such a topK finding algorithm using > RankingMetrics ? > > > On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng wrote: > > > Let's narrow the context from matrix factorization to recommendation > > via ALS. It adds extra complexity if we treat it as a multi-class > > classification problem. ALS only outputs a single value for each > > prediction, which is hard to convert to probability distribution over > > the 5 rating levels. Treating it as a binary classification problem or > > a ranking problem does make sense. The RankingMetricc is in master. > > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC > > should be good to add as well. -Xiangrui > > > > > > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das > > > wrote: > > > Hi, > > > > > > In the current factorization flow, we cross validate on the test > dataset > > > using the RMSE number but there are some other measures which are worth > > > looking into. > > > > > > If we consider the problem as a regression problem and the ratings 1-5 > > are > > > considered as 5 classes, it is possible to generate a confusion matrix > > > using MultiClassMetrics.scala > > > > > > If the ratings are only 0/1 (like from the spotify demo from spark > > summit) > > > then it is possible to use Binary Classification Metrices to come up > with > > > the ROC curve... > > > > > > For topK user/products we should also look into prec@k and pdcg@k as > the > > > metric.. > > > > > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in > > > examples.MovielensALS along with RMSE ? > > > > > > Thanks. > > > Deb > > >
Re: matrix factorization cross validation
Is there an example of how to use RankingMetrics ? Let's take the user, document example...we get user x topic and document x topic matrices as the model... Now for each user, we can generate topK document by doing a sort on (1 x topic)dot(topic x document) and picking topK... Is it possible to validate such a topK finding algorithm using RankingMetrics ? On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng wrote: > Let's narrow the context from matrix factorization to recommendation > via ALS. It adds extra complexity if we treat it as a multi-class > classification problem. ALS only outputs a single value for each > prediction, which is hard to convert to probability distribution over > the 5 rating levels. Treating it as a binary classification problem or > a ranking problem does make sense. The RankingMetricc is in master. > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC > should be good to add as well. -Xiangrui > > > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das > wrote: > > Hi, > > > > In the current factorization flow, we cross validate on the test dataset > > using the RMSE number but there are some other measures which are worth > > looking into. > > > > If we consider the problem as a regression problem and the ratings 1-5 > are > > considered as 5 classes, it is possible to generate a confusion matrix > > using MultiClassMetrics.scala > > > > If the ratings are only 0/1 (like from the spotify demo from spark > summit) > > then it is possible to use Binary Classification Metrices to come up with > > the ROC curve... > > > > For topK user/products we should also look into prec@k and pdcg@k as the > > metric.. > > > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in > > examples.MovielensALS along with RMSE ? > > > > Thanks. > > Deb >
Re: matrix factorization cross validation
Makes sense for the binary and ranking problem but for example linear regression for multi-class also optimizes on RMSE but we still measure the prediction efficiency using some measure on confusion matrix...Is not the same idea should hold for ALS as well ? On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng wrote: > Let's narrow the context from matrix factorization to recommendation > via ALS. It adds extra complexity if we treat it as a multi-class > classification problem. ALS only outputs a single value for each > prediction, which is hard to convert to probability distribution over > the 5 rating levels. Treating it as a binary classification problem or > a ranking problem does make sense. The RankingMetricc is in master. > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC > should be good to add as well. -Xiangrui > > > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das > wrote: > > Hi, > > > > In the current factorization flow, we cross validate on the test dataset > > using the RMSE number but there are some other measures which are worth > > looking into. > > > > If we consider the problem as a regression problem and the ratings 1-5 > are > > considered as 5 classes, it is possible to generate a confusion matrix > > using MultiClassMetrics.scala > > > > If the ratings are only 0/1 (like from the spotify demo from spark > summit) > > then it is possible to use Binary Classification Metrices to come up with > > the ROC curve... > > > > For topK user/products we should also look into prec@k and pdcg@k as the > > metric.. > > > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in > > examples.MovielensALS along with RMSE ? > > > > Thanks. > > Deb >
Re: matrix factorization cross validation
Let's narrow the context from matrix factorization to recommendation via ALS. It adds extra complexity if we treat it as a multi-class classification problem. ALS only outputs a single value for each prediction, which is hard to convert to probability distribution over the 5 rating levels. Treating it as a binary classification problem or a ranking problem does make sense. The RankingMetricc is in master. Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC should be good to add as well. -Xiangrui On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das wrote: > Hi, > > In the current factorization flow, we cross validate on the test dataset > using the RMSE number but there are some other measures which are worth > looking into. > > If we consider the problem as a regression problem and the ratings 1-5 are > considered as 5 classes, it is possible to generate a confusion matrix > using MultiClassMetrics.scala > > If the ratings are only 0/1 (like from the spotify demo from spark summit) > then it is possible to use Binary Classification Metrices to come up with > the ROC curve... > > For topK user/products we should also look into prec@k and pdcg@k as the > metric.. > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in > examples.MovielensALS along with RMSE ? > > Thanks. > Deb - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org