[
https://issues.apache.org/jira/browse/SPARK-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236796#comment-15236796
]
Nick Pentreath edited comment on SPARK-13857 at 4/12/16 9:19 AM:
-----------------------------------------------------------------
[~mengxr] [~josephkb]
In an ideal world, this is what train-validation split with ALS with ranking
evaluation would look like:
{code}
// Prepare training and test data.
val ratings = ...
val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2))
// set up ALS with top-k prediction
val als = new ALS()
.setMaxIter(5)
.setImplicitPrefs(true)
.setK(10)
.setTopKInputCol("user")
.setTopKOutputCol("topk")
// build param grid
val paramGrid = new ParamGridBuilder()
.addGrid(als.regParam, Array(0.01, 0.05, 0.1))
.addGrid(als.alpha, Array(1.0, 10.0, 20.0))
.build()
// ranking evaluator with appropriate prediction column
val evaluator = new RankingEvaluator()
.setPredictionCol("topk")
.setMetricName("mapk")
.setK(10)
.setLabelCol("actual")
val trainValidationSplit = new TrainValidationSplit()
.setEstimator(als)
.setEvaluator(evaluator)
.setEstimatorParamMaps(paramGrid)
// 80% of the data will be used for training and the remaining 20% for
validation.
.setTrainRatio(0.8)
// Run train validation split, and choose the best set of parameters.
val model = trainValidationSplit.fit(training)
// Make predictions on test data. model is the model with combination of
parameters
// that performed best.
model.transform(test)
.select("user", "actual", "topk")
.show()
{code}
This issue is, the input dataset to {{fit}} ALS is DF of {{(userId, itemId,
rating)}} rows. The input to {{transform}} with the top-k option enabled is a
DF of {{userId}} rows, while the input to {{evaluate}} is a DF of {{(userId,
actual)}} rows, where {{actual}} is an array of ground truth item ids {{(id1,
id2, ...)}}. So it doesn't work out the box.
I see three solutions:
# have {{RankingEvaluator}} and/or the cross-validation classes handle this in
some generic way (it would be good to understand how other ranking evaluation
use cases could look in order to also support them).
# have {{ALS}} handle it in {{transform}} - perhaps an option to output a
{{topk}} column and an {{actual}} column. This would require that the input DF
to {{transform}} with the top-k option is in the same form as for {{transform}}
normally. It would require a distinct on the {{userId}} column to only predict
for unique user ids, and may be a bit convoluted to make it work.
# have specialized versions of {{TrainValidationSplit}} and {{CrossValidator}}
to handle the recommendation case.
#3 is not actually as crazy as it may sound - since the recommendation case is
a little different (the same might be the case for say learning-to-rank on
queries etc), and even the way to split the dataset into train,test is
different (e.g. in recommender systems, often data is sampled by userid, such
as split a fraction of ratings for each user into the train and test sets, etc).
was (Author: mlnick):
[~mengxr] [~josephkb]
In an ideal world, this is what train-validation split with ALS would look like:
{code}
// Prepare training and test data.
val ratings = ...
val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2))
// set up ALS with top-k prediction
val als = new ALS()
.setMaxIter(5)
.setImplicitPrefs(true)
.setK(10)
.setTopKInputCol("user")
.setTopKOutputCol("topk")
// build param grid
val paramGrid = new ParamGridBuilder()
.addGrid(als.regParam, Array(0.01, 0.05, 0.1))
.addGrid(als.alpha, Array(1.0, 10.0, 20.0))
.build()
// ranking evaluator with appropriate prediction column
val evaluator = new RankingEvaluator()
.setPredictionCol("topk")
.setMetricName("mapk")
.setK(10)
.setLabelCol("actual")
val trainValidationSplit = new TrainValidationSplit()
.setEstimator(als)
.setEvaluator(evaluator)
.setEstimatorParamMaps(paramGrid)
// 80% of the data will be used for training and the remaining 20% for
validation.
.setTrainRatio(0.8)
// Run train validation split, and choose the best set of parameters.
val model = trainValidationSplit.fit(training)
// Make predictions on test data. model is the model with combination of
parameters
// that performed best.
model.transform(test)
.select("user", "actual", "topk")
.show()
{code}
> Feature parity for ALS ML with MLLIB
> ------------------------------------
>
> Key: SPARK-13857
> URL: https://issues.apache.org/jira/browse/SPARK-13857
> Project: Spark
> Issue Type: Sub-task
> Components: ML
> Reporter: Nick Pentreath
> Assignee: Nick Pentreath
>
> Currently {{mllib.recommendation.MatrixFactorizationModel}} has methods
> {{recommendProducts/recommendUsers}} for recommending top K to a given user /
> item, as well as {{recommendProductsForUsers/recommendUsersForProducts}} to
> recommend top K across all users/items.
> Additionally, SPARK-10802 is for adding the ability to do
> {{recommendProductsForUsers}} for a subset of users (or vice versa).
> Look at exposing or porting (as appropriate) these methods to ALS in ML.
> Investigate if efficiency can be improved at the same time (see SPARK-11968).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
