[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85791/ Test PASSed. ---

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #85791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85791/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #85791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85791/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85786/ Test FAILed. ---

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #85786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85786/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-07 Thread sameeragarwal
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/19080 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #85786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85786/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85715/ Test PASSed. ---

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #85715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85715/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #85715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85715/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2018-01-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84325/ Test PASSed. ---

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #84325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84325/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #84325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84325/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-11-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82802/ Test PASSed. ---

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #82802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82802/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #82802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82802/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-10-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 cc @rxin @JoshRosen @liancheng @sameeragarwal @gatorsmile @brkyvz any more comments? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 also cc @rxin , to support the "pre-shuffle" feature for data source v2, I need to create similar `Distribution` and `Partitioning` interfaces in the data source package. However, the current

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 so my whole point of view is, co-partition is a really tricky requirement, and it's really hard to implicitly guarantee it during shuffle planning. We should have a weaker guarantee(same number

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 > Both sides will satisfy the required distribution of the join This is not true now. After this PR, join has a stricter distribution requirement called `HashPartitionedDistribution`, so

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/19080 Have a question after reading the new approach. Let's say that we have a join like `T1 JOIN T2 on T1.a = T2.a`. Also `T1` is hash partitioned by the value of `T1.a` and it has 10 partitions, and `T2`

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19080 cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81264/ Test PASSed. ---

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19080 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #81264 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81264/testReport)** for PR 19080 at commit

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19080 also cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #19080: [SPARK-21865][SQL] simplify the distribution semantic of...

2017-08-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19080 **[Test build #81264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81264/testReport)** for PR 19080 at commit