[ https://issues.apache.org/jira/browse/SPARK-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291539#comment-14291539 ]
Apache Spark commented on SPARK-5406: ------------------------------------- User 'hhbyyh' has created a pull request for this issue: https://github.com/apache/spark/pull/4200 > LocalLAPACK mode in RowMatrix.computeSVD should have much smaller upper bound > ----------------------------------------------------------------------------- > > Key: SPARK-5406 > URL: https://issues.apache.org/jira/browse/SPARK-5406 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.2.0 > Environment: centos, others should be similar > Reporter: yuhao yang > Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > In RowMatrix.computeSVD, under LocalLAPACK mode, the code would invoke > brzSvd. Yet breeze svd for dense matrix has latent constraint. In it's > implementation: > val workSize = ( 3 > * scala.math.min(m, n) > * scala.math.min(m, n) > + scala.math.max(scala.math.max(m, n), 4 * scala.math.min(m, n) > * scala.math.min(m, n) + 4 * scala.math.min(m, n)) > ) > val work = new Array[Double](workSize) > as a result, column num must satisfy 7 * n * n + 4 * n < Int.MaxValue > thus, n < 17515. > This jira is only the first step. If possbile, I hope spark can handle matrix > computation up to 80K * 80K. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org