Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
thanks for your great work! merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fe
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@cloud-fan could you please review this again?
I addressed your comments. After rebasing, the performance issue has been
solved.
---
If your project is set up for it, you can reply to this email
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65930/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65930 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65930/consoleFull)**
for PR 13680 at commit
[`2ef6e3b`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65920/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65920 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65920/consoleFull)**
for PR 13680 at commit
[`2ef6e3b`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65930 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65930/consoleFull)**
for PR 13680 at commit
[`2ef6e3b`](https://github.com/apache/spark/commit/2
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishe
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65920 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65920/consoleFull)**
for PR 13680 at commit
[`2ef6e3b`](https://github.com/apache/spark/commit/2
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
When I rerun the benchmark, I got the performance regression only for
`Write UnsafeArrayData with double`. I am investigating reasons.
---
If your project is set up for it, you can reply to this emai
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65676/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65676 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65676/consoleFull)**
for PR 13680 at commit
[`f19020f`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65676 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65676/consoleFull)**
for PR 13680 at commit
[`f19020f`](https://github.com/apache/spark/commit/f
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65657/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65657 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65657/consoleFull)**
for PR 13680 at commit
[`11d930a`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65657 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65657/consoleFull)**
for PR 13680 at commit
[`11d930a`](https://github.com/apache/spark/commit/1
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65651/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65651 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65651/consoleFull)**
for PR 13680 at commit
[`1333247`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65651 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65651/consoleFull)**
for PR 13680 at commit
[`1333247`](https://github.com/apache/spark/commit/1
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65643/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65643 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65643/consoleFull)**
for PR 13680 at commit
[`813cea6`](https://github.com/apache/spark/commit/
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
Sorry for the delay, it's pretty close! I left some minor comments, thanks
for working on it!
BTW, can you rerun the benchmark and update your PR description? The word
alignment may have
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65643 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65643/consoleFull)**
for PR 13680 at commit
[`813cea6`](https://github.com/apache/spark/commit/8
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@cloud-fan would it be possible to review this? I think that I implemented
your suggestions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
ping @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@cloud-fan Could you please review this since all tests are passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65156/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65156 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65156/consoleFull)**
for PR 13680 at commit
[`813cea6`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #65156 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65156/consoleFull)**
for PR 13680 at commit
[`813cea6`](https://github.com/apache/spark/commit/8
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64121/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #64121 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64121/consoleFull)**
for PR 13680 at commit
[`f418f9c`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #64121 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64121/consoleFull)**
for PR 13680 at commit
[`f418f9c`](https://github.com/apache/spark/commit/f
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64006/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #64006 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64006/consoleFull)**
for PR 13680 at commit
[`cd28879`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #64006 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64006/consoleFull)**
for PR 13680 at commit
[`cd28879`](https://github.com/apache/spark/commit/c
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
You are right. I am addressing both.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
for 1, currently we only store offsets for var-length element, we should
store offset and length as the element content will be word-aligned.
---
If your project is set up for it, you can reply t
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@cloud-fan I see. I think that we have already done 1. I will address 2. Am
I correct?
1. in the `[offset or primitive values]`, we store the primitive values
directly, or store length and of
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
see https://github.com/apache/spark/pull/13680#discussion_r74798477
We'd like to change the format to:
[numElements] [null bits] [offset or primitive values] [values]
in the `[
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@cloud-fan @davies Are there additional things for me to do to merge this
PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your p
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@davies could you please review this again since I addressed comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project d
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63467/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63467 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63467/consoleFull)**
for PR 13680 at commit
[`d9a269d`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63467 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63467/consoleFull)**
for PR 13680 at commit
[`d9a269d`](https://github.com/apache/spark/commit/d
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63318/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63318 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63318/consoleFull)**
for PR 13680 at commit
[`87aca80`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63318 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63318/consoleFull)**
for PR 13680 at commit
[`87aca80`](https://github.com/apache/spark/commit/8
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63301/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63301 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63301/consoleFull)**
for PR 13680 at commit
[`7b4e819`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63301 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63301/consoleFull)**
for PR 13680 at commit
[`7b4e819`](https://github.com/apache/spark/commit/7
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63144/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63144 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63144/consoleFull)**
for PR 13680 at commit
[`04d2e4b`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #63144 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63144/consoleFull)**
for PR 13680 at commit
[`04d2e4b`](https://github.com/apache/spark/commit/0
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
cc @davies can you review this? thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
Thank you for letting us know the status. I did not know @davies is on
vacation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If yo
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
We need inputs from @davies , but unfortunately he is on vacation now...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@cloud-fan How about status of other reviewers?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featu
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
I double-checked benchmark results, and corrected results in `Write
UnsafeArrayData`. Now, all of them are up to date.
---
If your project is set up for it, you can reply to this email and have your
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
yea LGTM. And please make sure the benchmark results are up to date, as the
code has changed a lot of times.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61977/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61977 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61977/consoleFull)**
for PR 13680 at commit
[`04d2e4b`](https://github.com/apache/spark/commit/
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
@cloud-fan , thank you for a lot of great review comments. This PR looks
much better than the first commit.
I updated the description of this PR. Is it fine with you?
---
If your project is set u
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61977 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61977/consoleFull)**
for PR 13680 at commit
[`04d2e4b`](https://github.com/apache/spark/commit/0
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61972/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61972 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61972/consoleFull)**
for PR 13680 at commit
[`04d2e4b`](https://github.com/apache/spark/commit/
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
the code LGTM. @kiszk can you improve the PR description to explain more
about the new format? And can you comment on the benchmark result in the
benchmark file with benchmark result before your P
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61972 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61972/consoleFull)**
for PR 13680 at commit
[`04d2e4b`](https://github.com/apache/spark/commit/0
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
LGTM except some minor comments, it's pretty close! One easy-to-ignore
comment: https://github.com/apache/spark/pull/13680/files#r69849567
---
If your project is set up for it, you can reply to t
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61917/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61917 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61917/consoleFull)**
for PR 13680 at commit
[`e6da751`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61917 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61917/consoleFull)**
for PR 13680 at commit
[`e6da751`](https://github.com/apache/spark/commit/e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13680
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61839/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61839 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61839/consoleFull)**
for PR 13680 at commit
[`65fca21`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13680
**[Test build #61839 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61839/consoleFull)**
for PR 13680 at commit
[`65fca21`](https://github.com/apache/spark/commit/6
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
I think using `negative` has some performance penalty, we need to call
`math.abs` when calculating the element length. What do you think?
---
If your project is set up for it, you can reply to th
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
To use negative is for ease of debugging. We could use positive value, too.
Is the positive value preferable to you?
> This is fine. We zero out the null field instead of leaving some random
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
> `(4-bit nullbits)[0110] [offset0] [-offset3] [-offset3] [offset3]`
Why negative the `offset3`? The old format negative it to indicate
nullability, but we have null bits now.
> i
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
Ah, you are right. Exactly, [this
code](https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscFunctionsSuite.scala#L150)
writes ```UTF8
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13680
The old unsafe array format also store offsets only, can we follow it to
handle this case? BTW, when you say `not written`, do you mean writing null?
---
If your project is set up for it, you can
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13680
I think that it is not easy to put ```[not written, use offset3]``` with
good performance. I am thinking about **two cases**.
In **case 1**, my assumptions are
* Do not initialize ```[offs
1 - 100 of 160 matches
Mail list logo