[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 thanks for your great work! merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @cloud-fan could you please review this again? I addressed your comments. After rebasing, the performance issue has been solved. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65930/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65930/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65920/ Test FAILed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65920/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65930/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65920/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-21 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 When I rerun the benchmark, I got the performance regression only for `Write UnsafeArrayData with double`. I am investigating reasons. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65676/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65676 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65676/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65676 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65676/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65657/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65657/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65657/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65651/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65651/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65651/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65643/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65643/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 Sorry for the delay, it's pretty close! I left some minor comments, thanks for working on it! BTW, can you rerun the benchmark and update your PR description? The word alignment may have

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65643/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-18 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @cloud-fan would it be possible to review this? I think that I implemented your suggestions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-14 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 ping @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-09 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @cloud-fan Could you please review this since all tests are passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65156/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65156/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #65156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65156/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64121/ Test FAILed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #64121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64121/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #64121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64121/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64006/ Test FAILed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #64006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64006/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #64006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64006/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 You are right. I am addressing both. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 for 1, currently we only store offsets for var-length element, we should store offset and length as the element content will be word-aligned. --- If your project is set up for it, you can reply

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @cloud-fan I see. I think that we have already done 1. I will address 2. Am I correct? 1. in the `[offset or primitive values]`, we store the primitive values directly, or store length and

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 see https://github.com/apache/spark/pull/13680#discussion_r74798477 We'd like to change the format to: [numElements] [null bits] [offset or primitive values] [values] in the

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-17 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @cloud-fan @davies Are there additional things for me to do to merge this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @davies could you please review this again since I addressed comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63467/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63467/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63467/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63318/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63318/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63318/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63301/ Test FAILed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63301/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63301/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63144/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63144/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #63144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63144/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-02 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 cc @davies can you review this? thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-08-02 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-28 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 Thank you for letting us know the status. I did not know @davies is on vacation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 We need inputs from @davies , but unfortunately he is on vacation now... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-27 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @cloud-fan How about status of other reviewers? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 I double-checked benchmark results, and corrected results in `Write UnsafeArrayData`. Now, all of them are up to date. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 yea LGTM. And please make sure the benchmark results are up to date, as the code has changed a lot of times. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61977/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61977/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 @cloud-fan , thank you for a lot of great review comments. This PR looks much better than the first commit. I updated the description of this PR. Is it fine with you? --- If your project is set

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61977/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61972/ Test FAILed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61972 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61972/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 the code LGTM. @kiszk can you improve the PR description to explain more about the new format? And can you comment on the benchmark result in the benchmark file with benchmark result before your

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61972/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-07 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 LGTM except some minor comments, it's pretty close! One easy-to-ignore comment: https://github.com/apache/spark/pull/13680/files#r69849567 --- If your project is set up for it, you can reply to

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61917/ Test FAILed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61917/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61917/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61839/ Test PASSed. ---

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61839/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13680 **[Test build #61839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61839/consoleFull)** for PR 13680 at commit

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 I think using `negative` has some performance penalty, we need to call `math.abs` when calculating the element length. What do you think? --- If your project is set up for it, you can reply to

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 To use negative is for ease of debugging. We could use positive value, too. Is the positive value preferable to you? > This is fine. We zero out the null field instead of leaving some random

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 > `(4-bit nullbits)[0110] [offset0] [-offset3] [-offset3] [offset3]` Why negative the `offset3`? The old format negative it to indicate nullability, but we have null bits now. >

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 Ah, you are right. Exactly, [this code](https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscFunctionsSuite.scala#L150) writes

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13680 The old unsafe array format also store offsets only, can we follow it to handle this case? BTW, when you say `not written`, do you mean writing null? --- If your project is set up for it, you

[GitHub] spark issue #13680: [SPARK-15962][SQL] Introduce implementation with a dense...

2016-07-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13680 I think that it is not easy to put ```[not written, use offset3]``` with good performance. I am thinking about **two cases**. In **case 1**, my assumptions are * Do not initialize

  1   2   >