Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
@BryanCutler @wesm @cpcloud I filed a JIRA issue for decimal type support
[SPARK-21552](https://issues.apache.org/jira/browse/SPARK-21552) and sent a pr
for it as WIP #18754.
Let's move on there
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18655
LGTM, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wish
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79996/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79996 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79996/testReport)**
for PR 18655 at commit
[`b85dc23`](https://github.com/apache/spark/commit/b
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79995/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79995 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79995/testReport)**
for PR 18655 at commit
[`0bac10d`](https://github.com/apache/spark/commit/0
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79996 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79996/testReport)**
for PR 18655 at commit
[`b85dc23`](https://github.com/apache/spark/commit/b8
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79995 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79995/testReport)**
for PR 18655 at commit
[`0bac10d`](https://github.com/apache/spark/commit/0b
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79960/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79960 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79960/testReport)**
for PR 18655 at commit
[`19f3973`](https://github.com/apache/spark/commit/1
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79960 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79960/testReport)**
for PR 18655 at commit
[`19f3973`](https://github.com/apache/spark/commit/19
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wis
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79957/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79957 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79957/testReport)**
for PR 18655 at commit
[`19f3973`](https://github.com/apache/spark/commit/1
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79955/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79955 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79955/testReport)**
for PR 18655 at commit
[`5bbb46f`](https://github.com/apache/spark/commit/5
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79957 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79957/testReport)**
for PR 18655 at commit
[`19f3973`](https://github.com/apache/spark/commit/19
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79955 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79955/testReport)**
for PR 18655 at commit
[`5bbb46f`](https://github.com/apache/spark/commit/5b
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18655
+1 on holding off for `DecimalType` support
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18655
yes let leave decimal support for folllow-ups
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
@BryanCutler @wesm @cpcloud Thank you for reviewing this.
If the remaining issue here is only `DecimalType` support, I'd like to
separate it from this pr and merge this first to avoid duplicating
Github user wesm commented on the issue:
https://github.com/apache/spark/pull/18655
There are a bunch of open JIRAs about decimals in Arrow:
https://issues.apache.org/jira/issues/?filter=12334829&jql=project%20%3D%20ARROW%20AND%20status%20in%20(%22In%20Review%22%2C%20Open%2C%20%22In%20
Github user cpcloud commented on the issue:
https://github.com/apache/spark/pull/18655
Based on this code:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala#L429-L547
It looks like there are two types:
Github user wesm commented on the issue:
https://github.com/apache/spark/pull/18655
On DecimalType, I want to point out that we haven't hardened the memory
format and integration tests between Java<->C++ within Arrow. It would be great
if you could help with this -- we ran into a prob
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79798/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79798 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79798/testReport)**
for PR 18655 at commit
[`6fc4da0`](https://github.com/apache/spark/commit/6
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79798 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79798/testReport)**
for PR 18655 at commit
[`6fc4da0`](https://github.com/apache/spark/commit/6f
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79786/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79786 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79786/testReport)**
for PR 18655 at commit
[`7084b38`](https://github.com/apache/spark/commit/7
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79786 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79786/testReport)**
for PR 18655 at commit
[`7084b38`](https://github.com/apache/spark/commit/70
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79744/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79744 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79744/testReport)**
for PR 18655 at commit
[`a50a271`](https://github.com/apache/spark/commit/a
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79744 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79744/testReport)**
for PR 18655 at commit
[`a50a271`](https://github.com/apache/spark/commit/a5
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wis
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79741/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79741 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79741/testReport)**
for PR 18655 at commit
[`a50a271`](https://github.com/apache/spark/commit/a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79737/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79737 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79737/testReport)**
for PR 18655 at commit
[`b5988f9`](https://github.com/apache/spark/commit/b
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79741 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79741/testReport)**
for PR 18655 at commit
[`a50a271`](https://github.com/apache/spark/commit/a5
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79737 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79737/testReport)**
for PR 18655 at commit
[`b5988f9`](https://github.com/apache/spark/commit/b5
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
I see, I'll move files back to `arrow` package.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18655
> For ArrowConverters, I thought we can skip the intermediate
ArrowRecordBatch creation in ArrowConverters.toPayloadIterator(). What do you
think about that?
Ok, I see. By using `Arrow
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79698/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79698 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79698/testReport)**
for PR 18655 at commit
[`8ffedda`](https://github.com/apache/spark/commit/8
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
@BryanCutler I'd like to share the motivation of refactoring
`ArrowConverters` and `ColumnWriter`.
For `ColumnWriter`, at first I'd like to support complex types like
`ArrayType` and `Struct
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79698 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79698/testReport)**
for PR 18655 at commit
[`8ffedda`](https://github.com/apache/spark/commit/8f
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wis
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79696/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79696 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79696/testReport)**
for PR 18655 at commit
[`8ffedda`](https://github.com/apache/spark/commit/8
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79696 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79696/testReport)**
for PR 18655 at commit
[`8ffedda`](https://github.com/apache/spark/commit/8f
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
Thank you for your comments.
I agree that we should split this into smaller PRs. I'll push another
commit to remove `ArrowColumnVector` from this as soon as possible.
---
If your project is set
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18655
yea let's put `ArrowColumnVector` and its tests in a new PR and merge that
first.
`ArrowWriter` will also be used for pandas UDF, see
https://issues.apache.org/jira/browse/SPARK-21190 for
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18655
Thanks for this @ueshin. I agree with @kiszk that it would be easier to
review if you can split this into smaller PRs, maybe keep the additional type
support separate? I'm all for refactoring
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/18655
Good feature, but can we split this PR into smaller PRs for ease of review
since it looks large?
For example, since `ArrowColumnVector` is not used in refactored code, this
part can be moved to an
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18655
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79668/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79668 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79668/testReport)**
for PR 18655 at commit
[`58cd465`](https://github.com/apache/spark/commit/5
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18655
**[Test build #79668 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79668/testReport)**
for PR 18655 at commit
[`58cd465`](https://github.com/apache/spark/commit/58
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
cc @cloud-fan @BryanCutler
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wish
71 matches
Mail list logo
