yuhao yang created SPARK-5406: --------------------------------- Summary: LocalLAPACK mode in RowMatrix.computeSVD should have much smaller upper bound Key: SPARK-5406 URL: https://issues.apache.org/jira/browse/SPARK-5406 Project: Spark Issue Type: Bug Components: MLlib Affects Versions: 1.2.0 Environment: centos, others should be similar Reporter: yuhao yang Priority: Minor
In RowMatrix.computeSVD, under LocalLAPACK mode, the code would invoke brzSvd. Yet breeze svd for dense matrix has latent constraint. In it's implementation: val workSize = ( 3 * scala.math.min(m, n) * scala.math.min(m, n) + scala.math.max(scala.math.max(m, n), 4 * scala.math.min(m, n) * scala.math.min(m, n) + 4 * scala.math.min(m, n)) ) val work = new Array[Double](workSize) as a result, column num must satisfy 7 * n * n + 4 * n < Int.MaxValue thus, n < 17515. This jira is only the first step. If possbile, I hope spark can handle matrix computation up to 80K * 80K. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org