Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/17793#discussion_r114893509
--- Diff: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
---
@@ -910,26 +944,143 @@ object ALS extends DefaultParamsReadable[ALS] with
Logging {
private type FactorBlock = Array[Array[Float]]
/**
- * Out-link block that stores, for each dst (item/user) block, which src
(user/item) factors to
- * send. For example, outLinkBlock(0) contains the local indices (not
the original src IDs) of the
- * src factors in this block to send to dst block 0.
+ * A mapping of the columns of the items factor matrix that are needed
when calculating each row
+ * of the users factor matrix, and vice versa.
+ *
+ * Specifically, when calculating a user factor vector, since only those
columns of the items
+ * factor matrix that correspond to the items that that user has rated
are needed, we can avoid
+ * having to repeatedly copy the entire items factor matrix to each
worker later in the algorithm
+ * by precomputing these dependencies for all users, storing them in an
RDD of `OutBlock`s. The
+ * items' dependencies on the columns of the users factor matrix is
computed similarly.
+ *
+ * =Example=
+ *
+ * Using the example provided in the `InBlock` Scaladoc, `userOutBlocks`
would look like the
+ * following:
+ *
+ * {{{ userOutBlocks.collect() == Seq(
+ * 0 -> Array(Array(0, 1), Array(0, 1)),
+ * 1 -> Array(Array(0), Array(0))) }}}
+ *
+ * The data structure encodes the following information:
--- End diff --
This is all correct, but was still really confusing. Personally I think the
following is clearer, but if you don't then feel free to leave it out.
````scala
/**
* Each user block contains a subset of users in fixed, but typically
random order.
*
* User block 0 User block 1
* ________ _______
* | user12 | | user4 |
* | user5 | | user2 |
* | user33 | | |
* |________| |_______|
*
* Out block 0 Out block 1
*
* Array( Array(
* Array(0, 2), // item block 0 Array(0), // item block 0
* Array(1, 2), // item block 1 Array(0, 1), // item block 1
* Array(1)) // item block 2 Array()) // item block 2
*
* For outblocks, the index in the outer array correspond to the item
block. So the first inner
* array is item block 0, the second item block 1, and so on. The values
in each array correspond
* to the "local indices" of the user factors in this block that need to
be shipped to that item
* block. So for outblock 0, we know that user factors at index 0 and 2
must be shipped to item
* block 0. That means that the user factors for user12 and user33 need
to go to item block 0.
* And for outblock 1, we know that user4 must go to item block 0 and 1
and user2 must go to item
* block 1. None of the users in user block 1 need to go to item block 2.
*/
````
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
