[
https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li resolved SPARK-22271.
-----------------------------
Resolution: Fixed
Assignee: Huaxin Gao
Fix Version/s: 2.3.0
2.2.1
> Describe results in "null" for the value of "mean" of a numeric variable
> ------------------------------------------------------------------------
>
> Key: SPARK-22271
> URL: https://issues.apache.org/jira/browse/SPARK-22271
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Environment:
> Reporter: Shafique Jamal
> Assignee: Huaxin Gao
> Priority: Minor
> Fix For: 2.2.1, 2.3.0
>
> Attachments: decimalNumbers.zip
>
>
> Please excuse me if this issue was addressed already - I was unable to find
> it.
> Calling .describe().show() on my dataframe results in a value of null for the
> row "mean":
> {noformat}
> val foo = spark.read.parquet("decimalNumbers.parquet")
> foo.select(col("numericvariable")).describe().show()
> foo: org.apache.spark.sql.DataFrame = [numericvariable: decimal(38,32)]
> +-------+--------------------+
> |summary| numericvariable|
> +-------+--------------------+
> | count| 299|
> | mean| null|
> | stddev| 0.2376438793946738|
> | min|0.037815489727642...|
> | max|2.138189366554511...|
> {noformat}
> But all of the rows for this seem ok (I can attache a parquet file). When I
> round the column, however, all is fine:
> {noformat}
> foo.select(bround(col("numericvariable"), 31)).describe().show()
> +-------+---------------------------+
> |summary|bround(numericvariable, 31)|
> +-------+---------------------------+
> | count| 299|
> | mean| 0.139522503183236...|
> | stddev| 0.2376438793946738|
> | min| 0.037815489727642...|
> | max| 2.138189366554511...|
> +-------+---------------------------+
> {noformat}
> Rounding using 32 gives null also though.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
