[
https://issues.apache.org/jira/browse/SPARK-33487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-33487:
---------------------------------
Priority: Minor (was: Major)
> Let ML ALS recommend for BOTH subsets - users and items
> -------------------------------------------------------
>
> Key: SPARK-33487
> URL: https://issues.apache.org/jira/browse/SPARK-33487
> Project: Spark
> Issue Type: New Feature
> Components: ML
> Affects Versions: 3.0.1
> Reporter: Rose Aysina
> Priority: Minor
>
> Currently ALS in Spark ML supports next methods for getting recommendations:
> * {{recommendForAllUsers(numItems: Int): DataFrame}}
> * {{recommendForAllItems(numUsers: Int): DataFrame}}
> * {{recommendForUserSubset(dataset: Dataset[_], numItems: Int): DataFrame}}
> * {{recommendForItemSubset(dataset: Dataset[_], numUsers: Int): DataFrame}}
>
> *Feature request:* to add a method that recommends subset of items for subset
> of users, i.e. both users and items are selected from provided subsets.
> *Why it is important:* in real-time recommender systems you usually make
> predict for current users (that's why we need subset of users). And you can
> just recommend all items that you have, but only those who satisfy some
> business filters (that's why we need subset of items).
> *For example:* consider real-time news recommender system. Predict is done
> for small subset of users (say, for example, visitors for last minute), but
> it is not allowed to recommend old news or news not related to user country
> or etc, so at each predict we have some "white" list of items.
> So that's why it will be extremely useful to control *BOTH* for which users
> make recommendations *AND* which items include in these recommendations.
> *Related issues:* -SPARK-20679- , but there is just subsets either on users
> *OR* items.
> *What we do now:* just make additional filtering after
> {{recommendForUserSubset}} call, but this method has significant cost - we
> must receive all items recommendations, i.e. *{{numItems = # all available
> items}}* and then filter and only then select top-k among them.
> *Why it is bad:* usually subset of items allowed to recommend right now is
> much smaller than the amount of all seen items in an origin data (in my real
> dataset it is 220k vs 500).
> *Design:* I am sorry - I am not familiar with Spark internals so I offer
> solution based just on my human logic :)
> {code:scala}
> def recommendForUserItemSubsets(userDataset: Dataset[_],
> itemDataset: Dataset[_],
> numItems: Int): DataFrame = {
> val userFactorSubset = getSourceFactorSubset(dataset, userFactors,
> $(userCol))
> val itemFactorSubset = getSourceFactorSubset(dataset, itemFactors,
> $(itemCol))
> recommendForAll(userFactorSubset, itemFactorSubset, $(userCol),
> $(itemCol), numItems, $(blockSize))
> }
> {code}
>
> I will be glad to receive some feedback, is it reasonable request or not and
> maybe more efficient workarounds.
>
> Thanks!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
