[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 @rdblue Could you please review the latest code? It has been greatly simplified. No change in SqlBase.g4 necessary. For multiple coalesce hints, the optimizer will only pick the leftmost hint. This behavior is similar to multiple coalesce/repartition calls in Dataset, except the rightmost call is the winner. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #93943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93943/testReport)** for PR 21911 at commit [`33f5598`](https://github.com/apache/spark/commit/33f55984a93efca4588aa048d0e5eb7b9a80e9be). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 @gatorsmile Oracle's [PARALLEL Hint](https://docs.oracle.com/en/database/oracle/oracle-database/18/sqlrf/Comments.html#GUID-D25225CE-2DCE-4D9F-8E82-401839690A6E) is the closest I can find. And [SET CURRENT DEGREE](https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/sqlref/src/tpc/db2z_sql_setcurrentdegree.html) for parallel processing in DB2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21911 @jzhuge I knew all the DBMS have the hints. Do you know whether any system has a hint like `Coalesce`? Could you check the systems that have sophisticated hints like Oracle and DB2? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 Exploring a suggestion brought to me offline: make this hint a SELECT hint, not an INSERT hint. This way, the hint can be applied to subqueries. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 @gatorsmile Spark SQL Hint models after Hive SQL Hint. Impala also has hints: https://www.cloudera.com/documentation/enterprise/5-13-x/topics/impala_hints.html#hints. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21911 @jzhuge, I think it is confusing that this hint exposes the `shuffle` boolean flag. The Spark API makes a clear distinction between `repartition` and `coalesce` where `coalesce` means that Spark won't add a shuffle. Adding the boolean here would allow users to coalesce with a shuffle (repartition) or repartition without a shuffle (coalesce). I can't think of a good reason to do that. The reason why there is a boolean `shuffle` param is to use the same plan node internally, but I don't think that detail should be leaked into SQL hints. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21911 @jzhuge Do you know any other systems that have similar things? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21911 + @rdblue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93752/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #93752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93752/testReport)** for PR 21911 at commit [`4baa2c4`](https://github.com/apache/spark/commit/4baa2c43b2338ceb68c434a9e854bc0915cf8611). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` class ResolveCoalesceHints(conf: SQLConf) extends Rule[LogicalPlan] ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #93752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93752/testReport)** for PR 21911 at commit [`4baa2c4`](https://github.com/apache/spark/commit/4baa2c43b2338ceb68c434a9e854bc0915cf8611). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21911 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries
Github user holdensmagicalunicorn commented on the issue: https://github.com/apache/spark/pull/21911 @jzhuge, thanks! I am a bot who has found some folks who might be able to help with the review:@rxin, @gatorsmile and @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
