[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126451#comment-17126451
]
Hyukjin Kwon commented on SPARK-28624:
--
Reverted at
https://github.com/apache/spark/commit/a8266e44d464c31e37e4f7dbeb9210102ebef7c3
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png, collect
> make_date.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125548#comment-17125548
]
Apache Spark commented on SPARK-28624:
--
User 'MaxGekk' has created a pull request for this issue:
https://github.com/apache/spark/pull/28723
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Assignee: Maxim Gekk
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png, collect
> make_date.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125346#comment-17125346
]
Apache Spark commented on SPARK-28624:
--
User 'MaxGekk' has created a pull request for this issue:
https://github.com/apache/spark/pull/28721
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png, collect
> make_date.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125345#comment-17125345
]
Apache Spark commented on SPARK-28624:
--
User 'MaxGekk' has created a pull request for this issue:
https://github.com/apache/spark/pull/28721
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png, collect
> make_date.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080312#comment-17080312
]
Maxim Gekk commented on SPARK-28624:
toJavaDate is implemented differently in the master
[https://github.com/apache/spark/blob/e2d9399602d485eae94cd530d134ebab336e9e9b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L129-L132]
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png, collect
> make_date.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080303#comment-17080303
]
Yuming Wang commented on SPARK-28624:
-
This is because of {{toJavaDate}}:
https://github.com/apache/spark/blob/bb295d80e37440e6ee67a00bc09df2e9ff6e4e46/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L107-L116
{noformat}
scala> val d1 = java.time.LocalDate.ofEpochDay(-735525)
d1: java.time.LocalDate = -0044-03-15
scala> java.sql.Date.valueOf(d1)
res12: java.sql.Date = 0045-03-15
{noformat}
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png, collect
> make_date.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072694#comment-17072694
]
Yuming Wang commented on SPARK-28624:
-
[~maxgekk] collect() also returns incorrect result:
!collect make_date.png!
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png, collect
> make_date.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900090#comment-16900090
]
Maxim Gekk commented on SPARK-28624:
Just in case, the issue exists for negative years storing to Hive tables only.
Such years are out of valid range for DATE/TIMESTAMP types.
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900083#comment-16900083
]
Maxim Gekk commented on SPARK-28624:
The problem is in storing values of DATE type to a hive table. The values (days
since epoch) are converted to java.sql.Date, and after that serialized to
string by hive-exec *using Date.toString() which is wrong*, see the attached
screenshot.
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
> Attachments: Screen Shot 2019-08-05 at 18.19.39.png
>
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900059#comment-16900059
]
Maxim Gekk commented on SPARK-28624:
[~yumwang] Thanks. I will take a look at the issue.
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table
[
https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900031#comment-16900031
]
Yuming Wang commented on SPARK-28624:
-
cc [~maxgekk]
> make_date is inconsistent when reading from table
> -
>
> Key: SPARK-28624
> URL: https://issues.apache.org/jira/browse/SPARK-28624
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:sql}
> spark-sql> create table test_make_date as select make_date(-44, 3, 15) as d;
> spark-sql> select d, make_date(-44, 3, 15) from test_make_date;
> 0045-03-15-0044-03-15
> spark-sql>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
