[
https://issues.apache.org/jira/browse/SPARK-15660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-15660:
------------------------------------
Assignee: (was: Apache Spark)
> RDD and Dataset should show the consistent value for variance/stdev.
> --------------------------------------------------------------------
>
> Key: SPARK-15660
> URL: https://issues.apache.org/jira/browse/SPARK-15660
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Dongjoon Hyun
>
> In Spark-11490, `variance/stdev` are redefined as the **sample**
> `variance/stdev` instead of population ones.
> This issue addresses the only remaining legacy in RDD. This may cause
> breaking changes, but we had better be consistent in Spark 2.0 if possible.
> {code}
> scala> val rdd = sc.parallelize(Seq(1.0, 2.0, 3.0))
> rdd: org.apache.spark.rdd.RDD[Double] = ParallelCollectionRDD[0] at
> parallelize at <console>:24
> scala> rdd.stdev
> res0: Double = 0.816496580927726
>
> scala> rdd.toDS().describe().show()
> 16/05/30 22:20:12 WARN ObjectStore: Version information not found in
> metastore. hive.metastore.schema.verification is not enabled so recording the
> schema version 1.2.0
> 16/05/30 22:20:12 WARN ObjectStore: Failed to get database default, returning
> NoSuchObjectException
> +-------+-----+
> |summary|value|
> +-------+-----+
> | count| 3|
> | mean| 2.0|
> | stddev| 1.0|
> | min| 1.0|
> | max| 3.0|
> +-------+-----+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
