[
https://issues.apache.org/jira/browse/SPARK-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326828#comment-14326828
]
Xiangrui Meng commented on SPARK-5905:
--------------------------------------
1. If people say `an m x n matrix`, `m` should mean the number of rows and `n`
the number of columns. The notation is quite standard in linear algebra, as
used in the book `Matrix Computations`. For embedding, people may use the
notation `n x d`, where `n` is the number of rows and `d` the number of
columns. But in neither case, the first variable indicates the number of
columns.
2. This is not a hard requirement. We should update the user guide.
3. Yes, we should mention it in the doc.
> Improve RowMatrix user guide and doc.
> -------------------------------------
>
> Key: SPARK-5905
> URL: https://issues.apache.org/jira/browse/SPARK-5905
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, MLlib
> Affects Versions: 1.3.0
> Reporter: Xiangrui Meng
> Priority: Minor
>
> From mbofb's comment in PR https://github.com/apache/spark/pull/4680:
> {code}
> The description of RowMatrix.computeSVD and
> mllib-dimensionality-reduction.html should be more precise/explicit regarding
> the m x n matrix. In the current description I would conclude that n refers
> to the rows. According to
> http://math.stackexchange.com/questions/191711/how-many-rows-and-columns-are-in-an-m-x-n-matrix
> this way of describing a matrix is only used in particular domains. I as a
> reader interested on applying SVD would rather prefer the more common m x n
> way of rows x columns (e.g.
> http://en.wikipedia.org/wiki/Matrix_%28mathematics%29 ) which is also used in
> http://en.wikipedia.org/wiki/Latent_semantic_analysis (and also within the
> ARPACK manual:
> “
> N Integer. (INPUT) - Dimension of the eigenproblem.
> NEV Integer. (INPUT) - Number of eigenvalues of OP to be computed. 0 < NEV <
> N.
> NCV Integer. (INPUT) - Number of columns of the matrix V (less than or equal
> to N).
> “
> ).
> description of RowMatrix.computeSVD and mllib-dimensionality-reduction.html:
> "We assume n is smaller than m." Is this just a recommendation or a hard
> requirement. This condition seems not to be checked and causing an
> IllegalArgumentException – the processing finishes even though the vectors
> have a higher dimension than the number of vectors.
> description of RowMatrix. computePrincipalComponents or RowMatrix in general:
> I got a Exception.
> java.lang.IllegalArgumentException: Argument with more than 65535 cols:
> 7949273
> at
> org.apache.spark.mllib.linalg.distributed.RowMatrix.checkNumColumns(RowMatrix.scala:131)
> at
> org.apache.spark.mllib.linalg.distributed.RowMatrix.computeCovariance(RowMatrix.scala:318)
> at
> org.apache.spark.mllib.linalg.distributed.RowMatrix.computePrincipalComponents(RowMatrix.scala:373)
> This 65535 cols restriction would be nice to be written in the doc (if this
> still applies in 1.3).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
