Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84070/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84070 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84070/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84070 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84070/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84062/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84062 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84062/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84062 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84062/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84048/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84048 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84048/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84044/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84044 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84044/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84048 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84048/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Well, I'd leave the config
`spark.sql.execution.pandas.respectSessionTimeZone` as it is for now to be safe
as @gatorsmile mentioned before.
And as we discussed in dev list, I'll update this to
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84044 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84044/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84020/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84020 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84020/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84020 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84020/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84013 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84013/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84013/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
Seems there is no explicit objection for dropping it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #84013 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84013/testReport)**
for PR 19607 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
Here - https://github.com/apache/spark/pull/19674#issuecomment-342336767 I
checked it lately.
---
-
To unsubscribe, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19607
BTW what's the pandas version we set up in jenkins? I'm afraid we don't
even have a test for old pandas versions.
---
-
To
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
FWIW, to me, I think it's possibly a breaking change but I believe it's
worth given the recent cool changes. +0 for bumping up anyway. Will join in the
mailing thread too.
---
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
I'll ask dev list if we can drop support old Pandas (<0.19.2), or what
version we should support if we can't.
---
-
To
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19607
I doubt if we can still support pandas 0.13.0 now...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
For Py4J, as a hard required dependency, its src-zip is in Spark partly to
support non-pip installation as well properly AFAIK. In this case, the place to
update seems
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
AllI know is here:
https://github.com/apache/spark/blob/aad2125475dcdeb4a0410392b6706511db17bac4/python/setup.py#L199-L205
and few places in documentation:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19607
@HyukjinKwon do you know where pyspark defines its python library
requirements?
---
-
To unsubscribe, e-mail:
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19607
Correct me if I'm wrong, but I think depending on what version of Pandas we
support this PR could be simplified a little to only use
`is_datetime64tz_dtype` that is in 0.19.2+. So it might be
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
> Do we need the config "spark.sql.execution.pandas.respectSessionTimeZone"?
I think we don't need this in this case. If this discussion lasts without a
conclusion, I think we should
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
The remaining discussions in this pr are:
- [ ] Do we need the config
`"spark.sql.execution.pandas.respectSessionTimeZone"`?
- @cloud-fan raised that we don't need the config.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83780/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83780 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83780/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83777/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83777 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83777/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83780 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83780/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83777 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83777/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83774/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83774 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83774/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83774 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83774/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83593/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83593 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83593/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83592/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83592 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83592/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83593 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83593/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83592 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83592/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83586/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83586 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83586/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83586 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83586/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83578/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83578 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83578/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83578 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83578/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
@wesm Thanks for the explanation!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user wesm commented on the issue:
https://github.com/apache/spark/pull/19607
Well, old versions of pandas are not supported at all. There have also been
a number of API evolutions around extension dtypes that make supporting pandas
0.18.x and lower challenging. We have dropped
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
@wesm I'd like to know if you have any specific reason to recommend the
version 0.19.2 or upper?
---
-
To unsubscribe, e-mail:
Github user wesm commented on the issue:
https://github.com/apache/spark/pull/19607
@jreback @cpcloud could use some input on how to maximize cross-pandas
version compatibility in this patch.
I am not sure I would recommend that Spark support pandas versions lower
than
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
My local is:
```
+---+---+
|idx| timestamp|
+---+---+
| 0|1969-01-01 01:01:01|
| 1|2012-02-02 02:02:02|
| 2|
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19607
(Will give a shot to reproduce this within tomorrow and leave my output.)
---
-
To unsubscribe, e-mail:
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
I have no idea about the reason but seems like there is a difference of
handling DST in `spark.createDataFrame(data, schema=schema)` between Jenkins
and my local environments.
The debug
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83475/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83475 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83475/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Hmm, I can't reproduce the current failure in my local, which seems to not
be related to old Pandas because the failure is on Python 3.4 and Pandas 0.19
(I guess). I tried the combination in my
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83475 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83475/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83474 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83474/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83474/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83472/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83472 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83472/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83474 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83474/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83473 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83473/testReport)**
for PR 19607 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83473/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83473 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83473/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83472 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83472/testReport)**
for PR 19607 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19607
>Aside: Could someone start preparing a patch that uses Arrow 0.8.x (where
a lot of issues that surfaced throughout this process have been fixed)?
@wesm I could start on this maybe in
Github user wesm commented on the issue:
https://github.com/apache/spark/pull/19607
Aside: Could someone start preparing a patch that uses Arrow 0.8.x (where a
lot of issues that surfaced throughout this process have been fixed)?
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83324/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83324 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83324/testReport)**
for PR 19607 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19607
**[Test build #83324 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83324/testReport)**
for PR 19607 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
1 - 100 of 107 matches
Mail list logo
