[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78806/ Test PASSed. ---

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78806/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78806/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78677/ Test PASSed. ---

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78677/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78677/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18416 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78636/ Test FAILed. ---

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78636/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78636/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18416 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78622/ Test FAILed. ---

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78622/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78622/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18416 @cloud-fan ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18416 We need to document clearly what will happen if we have duplicated elements in a catalyst array and we convert that array to set. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18416 ah i see your point. Then it makes sense to do so --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18416 The serialized field after `KryoEncoder` is a binary. So I guess it can't easily convert between Dataset and DataFrame. You can't convert the result of `collect_set` to a `Dataset[Set[T]]`.

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18416 why not use `KryoEncoder` for `Set`? Serializing `Set` to catalyst array loses the set property, and it just becomes a meaningless binary: most of the array methods are meaningless, e.g. get

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18416 cc @cloud-fan I'd like to hear your opinion about this `Set` support. Can you provide some insights? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18416 Currently I can't think of possible issues of serializing `Set` as array. But welcome comments to point any possible issues. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78581/ Test PASSed. ---

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78581/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78581/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18416 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78578/testReport)** for PR 18416 at commit

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78578/ Test FAILed. ---

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #78578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78578/testReport)** for PR 18416 at commit