[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 Thanks @gatorsmile for the review and commit! Thanks @rdblue @mgaido91 for the reviews! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21911 You can address the comment in the follow-up PR. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21911 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94185/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94185/testReport)** for PR 21911 at commit [`2dcab4b`](https://github.com/apache/spark/commit/2dcab4b491e38766889c9cb83b8bebd59aa507fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94173/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94173/testReport)** for PR 21911 at commit [`acda5e2`](https://github.com/apache/spark/commit/acda5e281f1e3e8fd87faf4c8582d39ea857d215). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94185/testReport)** for PR 21911 at commit [`2dcab4b`](https://github.com/apache/spark/commit/2dcab4b491e38766889c9cb83b8bebd59aa507fd). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94159/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94159/testReport)** for PR 21911 at commit [`c950b6e`](https://github.com/apache/spark/commit/c950b6e5e5f943155b296542238c08a412a35472). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94173/testReport)** for PR 21911 at commit [`acda5e2`](https://github.com/apache/spark/commit/acda5e281f1e3e8fd87faf4c8582d39ea857d215). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94159/testReport)** for PR 21911 at commit [`c950b6e`](https://github.com/apache/spark/commit/c950b6e5e5f943155b296542238c08a412a35472). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94108/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94108/testReport)** for PR 21911 at commit [`739aeb4`](https://github.com/apache/spark/commit/739aeb44e9b9bb15b74271e2b42fb3dfe6f1c8fe). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94108/testReport)** for PR 21911 at commit [`739aeb4`](https://github.com/apache/spark/commit/739aeb44e9b9bb15b74271e2b42fb3dfe6f1c8fe). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94047/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94047/testReport)** for PR 21911 at commit [`7ff6e4b`](https://github.com/apache/spark/commit/7ff6e4b113a5b34d77827e83d6152f7e9e482827). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21911
> Which test suite is a good place to add such end-to-end case?
org.apache.spark.sql.SQLQuerySuite might be the best place.
> Do we plan to support such call as df.hint("COALESCE", Seq(10))? Why not
just use df.coalesce(10)?
It also does not make sense to block it. Since this is a new hint, we
should check whether it works.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user jzhuge commented on the issue:
https://github.com/apache/spark/pull/21911
The new commit should have taken care of all review comments except the
following:
> Add an end-to-end test case.
Which test suite is a good place to add such end-to-end case?
> Add a unit test case by using Dataset.hint(name: String, parameters:
String*)
Found it not applicable after examining DataFrameHintSuite.âvarious hint
parameters" .
Do we plan to support such call as `df.hint("COALESCE", Seq(10))`? Why not
just use `df.coalesce(10)`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94047/testReport)** for PR 21911 at commit [`7ff6e4b`](https://github.com/apache/spark/commit/7ff6e4b113a5b34d77827e83d6152f7e9e482827). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 Thanks @gatorsmile. Will do. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21911 cc @maryannxue Please review it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21911 Generally, it looks good to me. Let us target this to the upcoming 2.4 release We need to improve the test coverage: - Add a test case when users specify multiple REPARTITION, COALESCE hints `... INSERT INTO s SELECT /*+ REPARTITION(100), COALESCE(500), COALESCE(10) */ * FROM t ...` - Add a test case when users specify BROADCAST and REPARTITION hints at the same time - Add an end-to-end test case. - Add a unit test case in ResolveHintsSuite.scala to verify the hint names are case insensitive. - Add more negative test cases. For example, `"SELECT /*+ COALESCE(1.0) */ * FROM t"` and `"SELECT /*+ COALESCE(3 + 4) */ * FROM t"` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21911 I'd like to fix the AnalysisError message and I noted one small nit in the tests. +1 when the AnalysisError message is fixed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93977/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #93977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93977/testReport)** for PR 21911 at commit [`33f5598`](https://github.com/apache/spark/commit/33f55984a93efca4588aa048d0e5eb7b9a80e9be). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #93977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93977/testReport)** for PR 21911 at commit [`33f5598`](https://github.com/apache/spark/commit/33f55984a93efca4588aa048d0e5eb7b9a80e9be). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21911 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 The test `StreamingInnerJoinSuite.stream stream inner join on windows - with watermark` might be flaky. It passed on my laptop. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93943/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #93943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93943/testReport)** for PR 21911 at commit [`33f5598`](https://github.com/apache/spark/commit/33f55984a93efca4588aa048d0e5eb7b9a80e9be). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
