[
https://issues.apache.org/jira/browse/SPARK-24175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-24175:
--------------------------------
Description:
The current Spark 2.4 migration guide is not well phrased. We should
1. State the before behavior
2. State the after behavior
3. Add a concrete example with code to illustrate.
For example:
Since Spark 2.4, Spark compares a DATE type with a TIMESTAMP type after
promotes both sides to TIMESTAMP. To set `false` to
`spark.sql.hive.compareDateTimestampInTimestamp` restores the previous
behavior. This option will be removed in Spark 3.0.
-->
In version 2.3 and earlier, Spark implicitly casts a timestamp column to date
type when comparing with a date column. In version 2.4 and later, Spark casts
the date column to timestamp type instead. As an example, "xxx" would result in
".." in Spark 2.3, and in Spark 2.4, the result would be "..."
was:The current Spark 2.4 migration guide is not well phrased.
> improve the Spark 2.4 migration guide
> -------------------------------------
>
> Key: SPARK-24175
> URL: https://issues.apache.org/jira/browse/SPARK-24175
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Wenchen Fan
> Priority: Major
>
> The current Spark 2.4 migration guide is not well phrased. We should
> 1. State the before behavior
> 2. State the after behavior
> 3. Add a concrete example with code to illustrate.
> For example:
> Since Spark 2.4, Spark compares a DATE type with a TIMESTAMP type after
> promotes both sides to TIMESTAMP. To set `false` to
> `spark.sql.hive.compareDateTimestampInTimestamp` restores the previous
> behavior. This option will be removed in Spark 3.0.
> -->
> In version 2.3 and earlier, Spark implicitly casts a timestamp column to date
> type when comparing with a date column. In version 2.4 and later, Spark casts
> the date column to timestamp type instead. As an example, "xxx" would result
> in ".." in Spark 2.3, and in Spark 2.4, the result would be "..."
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
