Github user debasish83 commented on the pull request:
https://github.com/apache/spark/pull/3098#issuecomment-62248133
@coderxiang @MLnick updated the PR with the steps mentioned above...I
experimented with recommendProducts API but to support map-reduce validation,
decided to use cartesian of user and product feature RDD...
It seems for MAP computation we can do further optimization and we don't
need rank of n predicted products but we can send the rank of the products that
are only in testSet...It needs change to RatingMatrix as well and most likely
should be a separate PR...Not sure if this optimization holds for prec@k and
ndcg@k as well...
Here are the results on MovieLens 1M dataset:
./bin/spark-submit --master spark://localhost:7077 --jars
~/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar
--total-executor-cores 4 --executor-memory 4g --driver-memory 1g --class
org.apache.spark.examples.mllib.MovieLensALS
./examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar --kryo --lambda 0.065
--validateRecommendation 1.0 hdfs://localhost:8020/sandbox/movielens/
Got 1000209 ratings from 6040 users on 3706 movies.
Training: 800392, test: 199817.
Test RMSE = 0.8492844823210867.
k 20 prec@k 0.05337860218615437
k 40 prec@k 0.05016561775422324
k 60 prec@k 0.04697471568952191
k 80 prec@k 0.044406260351109623
k 100 prec@k 0.04242630009937065
k 120 prec@k 0.040632935850723195
k 140 prec@k 0.0390656792693891
k 160 prec@k 0.037725654190129174
k 180 prec@k 0.03652699569393839
k 200 prec@k 0.035531632991056654
Test user MAP = 0.038054284339533735.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
