Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
gentle ping...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, o
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
Hi @rxin, do you mind if I ask what you do think about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
I tested the functions related with `add`/`sub`/`trunc` on date/timestamp
and It seems generally option 2 or option 3.
**DB2 (`TRUNC`)**
- input: `TimestampType`, output:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
I will be back after testing/looking into other databases soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project do
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
https://github.com/apache/spark/pull/14788#issuecomment-242906410 here is
my observation. It seems usually option 1 ir option 3. I can take a look deeper
if we want to follow this or be very sur
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14788
What do other databases do? Does date_add in other databases support
timestamps?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If you
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
Could I then go for option 1.?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14788
It would work with different names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enab
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14788
Does option 1 really work? Wouldn't the expressions have the same user
facing function names?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14788
I prefer option 1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
@rxin @hvanhovell @cloud-fan Could I please ask the behaviour we want here?
1. Separate expressions for both `DateType` and `TimestampType`
2. Returns `DateType` for `DateType` and `T
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66728/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66728 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66728/consoleFull)**
for PR 14788 at commit
[`cd78330`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66728 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66728/consoleFull)**
for PR 14788 at commit
[`cd78330`](https://github.com/apache/spark/commit/c
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66713/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66713 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66713/consoleFull)**
for PR 14788 at commit
[`ef67829`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66713 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66713/consoleFull)**
for PR 14788 at commit
[`ef67829`](https://github.com/apache/spark/commit/e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66712 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66712/consoleFull)**
for PR 14788 at commit
[`537fe88`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66712/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66712 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66712/consoleFull)**
for PR 14788 at commit
[`537fe88`](https://github.com/apache/spark/commit/5
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #3316 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3316/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #3316 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3316/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #3313 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3313/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #3313 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3313/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66626/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66626 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66626/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit/
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14788
Actually can we avoid renaming these expressions? I don't see the point to
rename DateSub to SubDays. It just makes it more annoying to link the user
facing API with the internal expressions.
---
If
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #66626 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66626/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit/8
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14788
LGTM - I'll merge as soon as tests complete successfully
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14788
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
@cloud-fan @hvanhovell Would there be other things I should double check
and take care of?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
Hi @cloud-fan and @hvanhovell , could I ask to take another look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65092/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #65092 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65092/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #65092 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65092/consoleFull)**
for PR 14788 at commit
[`8c50b2c`](https://github.com/apache/spark/commit/8
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65034/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #65034 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65034/consoleFull)**
for PR 14788 at commit
[`1998ce5`](https://github.com/apache/spark/commit/
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
Sure. Thanks for your close look. I will address the comments tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14788
Looks pretty good. I left a few more comments.
Could you also add a test for TruncDate that covers a day light saving
scenario?
---
If your project is set up for it, you can reply to th
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #65034 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65034/consoleFull)**
for PR 14788 at commit
[`1998ce5`](https://github.com/apache/spark/commit/1
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64934/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64934 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64934/consoleFull)**
for PR 14788 at commit
[`4eb6f54`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64934 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64934/consoleFull)**
for PR 14788 at commit
[`4eb6f54`](https://github.com/apache/spark/commit/4
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64921 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64921/consoleFull)**
for PR 14788 at commit
[`0040f35`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64921/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64921 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64921/consoleFull)**
for PR 14788 at commit
[`0040f35`](https://github.com/apache/spark/commit/0
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64918/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64918 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64918/consoleFull)**
for PR 14788 at commit
[`03bb6a1`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64918 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64918/consoleFull)**
for PR 14788 at commit
[`03bb6a1`](https://github.com/apache/spark/commit/0
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64917/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14788
For `date_add`, `date_sub`, `add_month`, I think we should support both
DateType and TimestampType, and the return type should depend on the input type.
For `last_day`, `first_day`, we sho
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
It seems it is up to how we define the behaviour. I will follow your
decision @cloud-fan.
My personal opinion is, support both types for `TRUNC` but not for
`LAST_DAY` and `NEXT_DAY`.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
I took a look for MySQL and PostgreSQL. It seems they are not really
consistent for `NEXT_DAY`, `LAST_DAY` and `TRUNC` but it seems the return types
are fixed in general (whether it is `DateType
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64524/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64524 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64524/consoleFull)**
for PR 14788 at commit
[`eee40b8`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64524 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64524/consoleFull)**
for PR 14788 at commit
[`eee40b8`](https://github.com/apache/spark/commit/e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64523/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64347/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14788
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64347 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64347/consoleFull)**
for PR 14788 at commit
[`777300e`](https://github.com/apache/spark/commit/
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14788
Oh, yes. right. I will address your comments tomorrow. Thank you
@hvanhovell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14788
@HyukjinKwon this looks pretty good.
I think we should also do this for `DateAdd`, `DateSub`, `LastDay` and
`NextDay` (the naming is a bit unfortunate for the first two though). We might
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14788
**[Test build #64347 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64347/consoleFull)**
for PR 14788 at commit
[`777300e`](https://github.com/apache/spark/commit/7
75 matches
Mail list logo
