[ https://issues.apache.org/jira/browse/SPARK-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179227#comment-15179227 ]
yuhao yang commented on SPARK-13639: ------------------------------------ We perhaps can find some other way for SPARK-13568. I just need to make sure the performance is acceptable. > Statistics.colStats(rdd).mean and variance should handle NaN in the input > vectors > --------------------------------------------------------------------------------- > > Key: SPARK-13639 > URL: https://issues.apache.org/jira/browse/SPARK-13639 > Project: Spark > Issue Type: Improvement > Components: MLlib > Reporter: yuhao yang > Priority: Trivial > > val denseData = Array( > Vectors.dense(3.8, 0.0, 1.8), > Vectors.dense(1.7, 0.9, 0.0), > Vectors.dense(Double.NaN, 0, 0.0) > ) > val rdd = sc.parallelize(denseData) > println(Statistics.colStats(rdd).mean) > [NaN,0.3,0.6] > This is just a proposal for discussion on how to handle the NaN value in the > vectors. We can ignore the NaN value in the computation or just output NaN as > it is now as a warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org