Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@ckadner The test function here try to similar this SQL function:
cast(from_unix_timestamp(cast("2016-03-13 01:59:59" as timestamp), "PST")
as string)
In the beginning, I want to
Github user ckadner commented on the issue:
https://github.com/apache/spark/pull/13652
@davies ...just to confirm the desired behavior of these two test cases
```from UTC timestamp``` and ```to UTC timestamp```. We are contending with 3
time zones:
* we are parsing
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@ckadner That could fix the flaky test, but hide the actual bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ckadner commented on the issue:
https://github.com/apache/spark/pull/13652
I (also) just created a [PR](https://github.com/apache/spark/pull/13785) to
fix the two test cases ```from UTC timestamp``` and ```to UTC timestap``` to
respect the time zones that are supposed to
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@lw-lin @robbinspg Thanks for report this, sent
https://github.com/apache/spark/pull/13784 to fix the bug.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user adrian-wang commented on the issue:
https://github.com/apache/spark/pull/13652
@lw-lin @robbinspg I submit a PR to fix the failure: #13783 . Thanks for
your information!
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user robbinspg commented on the issue:
https://github.com/apache/spark/pull/13652
Also failing here in the UK:
{noformat}
- to UTC timestamp *** FAILED ***
"2016-03-13 [02]:00:00.0" did not equal "2016-03-13 [10]:00:00.0"
(DateTimeUtilsSuite.scala:506)
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/13652
hi @davies, two tests still fail after this patch when I build locally:
```
- from UTC timestamp *** FAILED ***
"2016-03-13 0[2]:00:00.0" did not equal "2016-03-13 0[3]:00:00.0"
Github user 4e6 commented on the issue:
https://github.com/apache/spark/pull/13652
Awesome, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
Merged into 1.6 and 2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60775/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60775 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60775/consoleFull)**
for PR 13652 at commit
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
Since we have to cut a RC for 2.0, I will merge this into master and 2.0
after it pass tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@ckadner Had updated the PR to fallback to java.sql.Timestamp to do the
reverse lookup, that should have better resolution (especially when converting
a timestamp to another timezone).
The
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60775 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60775/consoleFull)**
for PR 13652 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60773/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60773 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60773/consoleFull)**
for PR 13652 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60773 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60773/consoleFull)**
for PR 13652 at commit
Github user ckadner commented on the issue:
https://github.com/apache/spark/pull/13652
I think this is a good "best effort" fix for the round-trip conversion
```int -> java.sql.Date -> int```.
It may also be worth testing the round-trip conversion ```java.sql.Date ->
int ->
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@JoshRosen How does this look now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #3119 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3119/consoleFull)**
for PR 13652 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60648/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60648 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60648/consoleFull)**
for PR 13652 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #3119 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3119/consoleFull)**
for PR 13652 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60647/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60647 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60647/consoleFull)**
for PR 13652 at commit
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@RussellSpitzer Unfortunately, most of the database do not support that
(cast a integer into date), same here. For parquet, we use the days since epoch
rather than using java.sql.Date.
---
If your
Github user davies commented on the issue:
https://github.com/apache/spark/pull/13652
@JoshRosen If any human time representation is involved, for example,
`2016-06-15 17:22:07.0`, a timezone could be used. In your SQL query, you are
comparing absolute timestamp again human time
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60648 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60648/consoleFull)**
for PR 13652 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60647 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60647/consoleFull)**
for PR 13652 at commit
Github user RussellSpitzer commented on the issue:
https://github.com/apache/spark/pull/13652
I would love to be able to just specify days since epoch rather than using
java.sql.Date
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/13652
It looks like there's some precedence for having the notion of separate
"session" and "server" timezones in MySQL and Postgres (as well as JDBC), so it
may be okay to have parsing of timestamp
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/13652
Question to check my conceptual understanding: if I don't use explicit
`java.sql.Date` functions or conversions and don't call any timezone functions,
etc., then should any time/date-related pure
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60607/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60607 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60607/consoleFull)**
for PR 13652 at commit
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/13652
I think a lot of my own confusion here stems from trying to reason through
what should happen if I take `java.sql.Date.valueOf("2016-01-01")` and convert
it into Spark's internal date
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #3110 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3110/consoleFull)**
for PR 13652 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60607 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60607/consoleFull)**
for PR 13652 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #3110 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3110/consoleFull)**
for PR 13652 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13652
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60511/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13652
**[Test build #60511 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60511/consoleFull)**
for PR 13652 at commit
47 matches
Mail list logo
