[
https://issues.apache.org/jira/browse/SPARK-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046264#comment-15046264
]
jinglei.chen commented on SPARK-10802:
--------------------------------------
recommendProductsForUsers(num: Int): RDD[(Int, Array[Rating])] spent much long
time, in final, it take error:User class threw exception:
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
hdfs://recomm-30-1.liepin.inc:9000/recommend/result/job/newb already exists
it run error on 1517459 users and 1190671 products, Every time an error, but
not error when the dataset is small
> Let ALS recommend for subset of data
> ------------------------------------
>
> Key: SPARK-10802
> URL: https://issues.apache.org/jira/browse/SPARK-10802
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.5.0
> Reporter: Tomasz Bartczak
> Priority: Minor
>
> Currently MatrixFactorizationModel allows to get recommendations for
> - single user
> - single product
> - all users
> - all products
> recommendation for all users/products do a cartesian join inside.
> It would be useful in some cases to get recommendations for subset of
> users/products by providing an RDD with which MatrixFactorizationModel could
> do an intersection before doing a cartesian join. This would make it much
> faster in situation where recommendations are needed only for subset of
> users/products, and when the subset is still too large to make it feasible to
> recommend one-by-one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
