Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22784#discussion_r228724594
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/feature/PCASuite.scala ---
@@ -54,4 +55,21 @@ class PCASuite extends SparkFunSuite with
MLlibTestSparkContext {
// check overflowing
assert(PCAUtil.memoryCost(40000, 60000) > Int.MaxValue)
}
+
+ test("number of features more than 65535") {
+ val rows = 10
+ val columns = 100000
+ val k = 5
+ val randomRDD = RandomRDDs.normalVectorRDD(sc, rows, columns, 0, 0)
+ val pca = new PCA(k).fit(randomRDD)
+ assert(pca.explainedVariance.size === 5)
+ assert(pca.pc.numRows === 100000 && pca.pc.numCols === 5)
+ // Eigen values should not be negative
+ assert(!pca.explainedVariance.values.exists(_ < 0))
+
+ // Norm of the principle component should be 1.0
--- End diff --
Nit: principle -> principal
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
