Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
Thanks @HyukjinKwon @ueshin and @viirya !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19459
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83761/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83761 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83761/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83761 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83761/testReport)**
for PR 19459 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19459
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83703/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83703 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83703/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83703 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83703/testReport)**
for PR 19459 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19459
Looks pretty solid. Will take a another look today (KST) and merge this one
in few days if there are no more comments and/or other committers are busy to
take a look and merge.
---
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
@ueshin @HyukjinKwon does this look ready to merge? cc @cloud-fan
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83647/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83647 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83647/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83647 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83647/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83635/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83635 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83635/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83635 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83635/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83579/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83579 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83579/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83579 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83579/testReport)**
for PR 19459 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19459
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83569/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83569 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83569/testReport)**
for PR 19459 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
I made [SPARK-22417](https://issues.apache.org/jira/browse/SPARK-22417) for
fixing reading from timestamps without arrow
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83233/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83233 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83233/testReport)**
for PR 19459 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19459
I think it is a bug, we should fix it first.
BTW I'm fine to upgrade arrow, just make sure we get everything we need at
the arrow version we wanna upgrade, then remove all the hacks at
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
After incorporating date and timestamp types for this, I had to refactor a
little to use `_create_batch` from serializers to make Arrow batches from
Columns even when the user doesn't specify
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83233 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83233/testReport)**
for PR 19459 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
@ueshin if possible I'd like to have #18664 merged first and then I can fix
this PR up if needed, thanks!
---
-
To
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19459
I guess this pr is almost ready to be merged.
I'd cc @gatorsmile @cloud-fan for another look.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83018/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83018 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83018/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83018 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83018/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83001/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83001 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83001/testReport)**
for PR 19459 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
Merged in PR from @ueshin and added case for when schema is a string single
datatype. In addition using a `StructType`, now this handles specifying the
schema with the following:
```
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #83001 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83001/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82894/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82894 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82894/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82894 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82894/testReport)**
for PR 19459 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
Yes, I meant to ask for some clarification from @ueshin
for https://github.com/apache/spark/pull/19459#discussion_r145034007
> Btw, do we also need to support schema like ['name',
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82866/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82866 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82866/testReport)**
for PR 19459 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19459
BTW, https://github.com/apache/spark/pull/19459#discussion_r145034007 looks
missed :).
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82866 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82866/testReport)**
for PR 19459 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
Thanks for reviewing @viirya !
I just had some followup questions at
https://github.com/apache/spark/pull/19459#discussion_r144930424 and
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19459
LGTM too but let me leave it to @ueshin just in case.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19459
LGTM with few minor comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82764/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82764 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82764/testReport)**
for PR 19459 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
Thanks for the reviews @ueshin and @HyukjinKwon! I added `to_arrow_schema`
conversion for when a schema is passed into `createDataFrame` and added some
new tests to verify it. Please take
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82764 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82764/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82601/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82601 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82601/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82601 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82601/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82573/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82573 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82573/testReport)**
for PR 19459 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82573 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82573/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82560/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82560 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82560/testReport)**
for PR 19459 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19459
Benchmarks for running in local mode 16 GB memory, i7-4800MQ CPU @ 2.70GHz
à 8 cores
using default Spark configuration
data is 10 columns of doubles with 100,000 rows
Code:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82560 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82560/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82559 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82559/testReport)**
for PR 19459 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19459
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82559/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19459
**[Test build #82559 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82559/testReport)**
for PR 19459 at commit
82 matches
Mail list logo
