[
https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183869#comment-15183869
]
Ian edited comment on SPARK-13731 at 3/8/16 7:29 PM:
-----------------------------------------------------
We saw SPARK-9076, which seemed addressing only how NaN value in RDD can be
handled(equality, ordering, ...),
but our concerned case is more on the another aspect and more fundamental about
the expressions that might produce NaN.
was (Author: ianlcsd):
The expression in select essentially defines a transformation from data
residing on storage or even another RDD. We are seeing that the transformation
result is now null for both NaN and Infinity.
We saw SPARK-9076, which seemed addressing only how NaN value in RDD can be
handled(equality, ordering, ...),
but our concerned case is more on the another aspect and more fundamental about
the expressions that might produce NaN.
> expression evaluation for NaN in select statement
> -------------------------------------------------
>
> Key: SPARK-13731
> URL: https://issues.apache.org/jira/browse/SPARK-13731
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Ian
>
> We are expecting that arithmetic expression a/b should be:
> 1. returning NaN if a=0 and b=0
> 2. returning Infinity if a=1 and b=0
> Is the expectation reasonable?
> The following is a simple test case snippet that reads from storage and
> evaluates arithmetic expressions in select.
> It is assuming org.apache.spark.sql.hive.execution.SQLQuerySuite:
> {code}
> test("Expression should be evaluated to Nan/Infinity in Select") {
> withTable("testNan") {
> withTempTable("src") {
> Seq((1d, 0d), (0d, 0d)).toDF().registerTempTable("src")
> sql("CREATE TABLE testNan(a double, b double) STORED AS PARQUET AS
> SELECT * FROM src")
> }
> checkAnswer(sql(
> """
> |SELECT a/b FROM testNan
> """.stripMargin),
> Seq(
> Row(Double.PositiveInfinity),
> Row(Double.NaN)
> )
> )
> }
> }
> == Physical Plan ==
> Project [(a#28 / b#29) AS _c0#30]
> +- Scan ParquetRelation: default.testnan[a#28,b#29] InputPaths:
> file:/private/var/folders/dy/19y6pfm92pj9s40mbs8xd9hm0000gp/T/warehouse--5b617080-e909-4812-90e8-63d2dd0aef5a/testnan
> == Results ==
> !== Correct Answer - 2 == == Spark Answer - 2 ==
> ![Infinity] [null]
> ![NaN] [null]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
