[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204654670 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54759/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204654669 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204654619 **[Test build #54759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54759/consoleFull)** for PR 12099 at commit [`9334fa9`](https://github.com/apache/spark/commit/9334fa9930222912cd847911eb04eb90da9253de). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204652575 Thank you, @rxin , @hvanhovell , @andrewor14 , and @DeepSparkBot . :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12099 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204650933 Thanks - going to merge this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204649298 **[Test build #2728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2728/consoleFull)** for PR 12099 at commit [`7b10a51`](https://github.com/apache/spark/commit/7b10a517e04c72235beb9cfa271dc540df75b763). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204642011 **[Test build #54759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54759/consoleFull)** for PR 12099 at commit [`9334fa9`](https://github.com/apache/spark/commit/9334fa9930222912cd847911eb04eb90da9253de). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58288294
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala
---
@@ -101,4 +101,19 @@ class HiveExplainSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
"Physical Plan should not contain Subquery since it's eliminated
by optimizer")
}
}
+
+ test("SPARK-14251: EXPLAIN CODEGEN command") {
--- End diff --
Thanks. Now, it's changed to **EXPLAIN CODEGEN command**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58288094
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala
---
@@ -101,4 +101,19 @@ class HiveExplainSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
"Physical Plan should not contain Subquery since it's eliminated
by optimizer")
}
}
+
+ test("SPARK-14251: EXPLAIN CODEGEN command") {
--- End diff --
yea i agree. it's best not to have the jira ticket here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204640856 **[Test build #2728 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2728/consoleFull)** for PR 12099 at commit [`7b10a51`](https://github.com/apache/spark/commit/7b10a517e04c72235beb9cfa271dc540df75b763). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204618176 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54736/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204618175 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204618122 **[Test build #54736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54736/consoleFull)** for PR 12099 at commit [`7b10a51`](https://github.com/apache/spark/commit/7b10a517e04c72235beb9cfa271dc540df75b763). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204603217 I updated the code according to @DeepSparkBot 's comments, and also rebased to the master since master branch is now Scala 2.11.8. It took sometime to test locally. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204603119 **[Test build #54736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54736/consoleFull)** for PR 12099 at commit [`7b10a51`](https://github.com/apache/spark/commit/7b10a517e04c72235beb9cfa271dc540df75b763). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204601255 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204601258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54717/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204600076 **[Test build #54717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54717/consoleFull)** for PR 12099 at commit [`497a93e`](https://github.com/apache/spark/commit/497a93ec34eb22655a9dabce5c08879fc9e1d30f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204591610 Oh, I see. Thank you, @andrewor14 ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user DeepSparkBot commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204589471 @andrewor14 Review complete. No major issues found. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user DeepSparkBot commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58275215
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -48,6 +48,25 @@ package object debug {
// scalastyle:on println
}
+ def codegenString(plan : SparkPlan): String = {
--- End diff --
**Detected minor style violation.**
Suggested improvement: remove whitespace before `:`
```
def codegenString(plan: SparkPlan): String = {
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user DeepSparkBot commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58275150
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
extends RunnableCommand {
// Run through the optimizer to generate the physical plan.
override def run(sqlContext: SQLContext): Seq[Row] = try {
// TODO in Hive, the "extended" ExplainCommand prints the AST as well,
and detailed properties.
val queryExecution = sqlContext.executePlan(logicalPlan)
-val outputString = if (extended) queryExecution.toString else
queryExecution.simpleString
-
+var outputString = if (extended) queryExecution.toString else
queryExecution.simpleString
+if (codegen) {
+ outputString = codegenString(queryExecution.executedPlan)
--- End diff --
**Detected unnecessary var.**
Suggested improvement:
```
val outputString
if (codegen) {
codegenString(queryExecution.executedPlan)
} else if (extended) {
queryExecution.toString
} else {
queryExecution.simpleString
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204587861 You need to say: deep-review this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204585811 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54713/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204585808 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204585214 **[Test build #54713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54713/consoleFull)** for PR 12099 at commit [`11efbbb`](https://github.com/apache/spark/commit/11efbbb587bfb844248e445e688b7f6819126c68). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204567123 One minor thing, why @DeepSparkBot does not visit this PR? Today, @DeepSparkBot came to my other PR. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204565788 Now, `EXPLAIN CODEGEN` is the logically same with `debugCodegen`. I updated the description of this PR, too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204564887 **[Test build #54717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54717/consoleFull)** for PR 12099 at commit [`497a93e`](https://github.com/apache/spark/commit/497a93ec34eb22655a9dabce5c08879fc9e1d30f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58265346
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
Thank you. I do it right now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58265199
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
yes explain codegen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58264981
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
Sure. I'll change that. By the way, I think you mean (1). Am I right?
(1) `EXPLAIN CODEGEN SELECT ~`
(2) `CODEGEN SELECT ~`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58264352
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
explain codegen should just dump the same thing debugCodegen dumps. Nothing
else.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58264076
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
As you know, I just implemented this because SPARK-14251 describes `explain
codegen select * ...`.
I thought SPARK-14251 aims to reduce the debugging efforts by showing the
full info in a single command.
If that is not a goal, we should change it.
You are the issue reporter of SPARK-14251. :)
If you want to do this in another way, I can change this right now. This is
the very right time. :)
Could you update
[SPARK-14251](https://issues.apache.org/jira/browse/SPARK-14251) for me before
making `CODEGEN` command?
Thank you, @rxin .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58260060
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
i like the sql -- just saying it'd be great to have different command
classes rather than having different flags in a single command.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204549678 **[Test build #54713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54713/consoleFull)** for PR 12099 at commit [`11efbbb`](https://github.com/apache/spark/commit/11efbbb587bfb844248e445e688b7f6819126c68). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204547543 Hi, @rxin . `debugCodegenString` is removed and `DebuggingSuite` is modified accordingly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58257745
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -81,29 +100,11 @@ package object debug {
* WholeStageCodegen subtree).
*/
def debugCodegen(): Unit = {
- debugPrint(debugCodegenString())
+ debugPrint(codegenString(query.queryExecution.executedPlan))
}
/** Visible for testing. */
-def debugCodegenString(): String = {
- val plan = query.queryExecution.executedPlan
- val codegenSubtrees = new
collection.mutable.HashSet[WholeStageCodegen]()
- plan transform {
-case s: WholeStageCodegen =>
- codegenSubtrees += s
- s
-case s => s
- }
- var output = s"Found ${codegenSubtrees.size} WholeStageCodegen
subtrees.\n"
- for ((s, i) <- codegenSubtrees.toSeq.zipWithIndex) {
-output += s"== Subtree ${i + 1} / ${codegenSubtrees.size} ==\n"
-output += s
-output += "\nGenerated code:\n"
-val (_, source) = s.doCodeGen()
-output += s"${CodeFormatter.format(source)}\n"
- }
- output
-}
+def debugCodegenString(): String =
codegenString(query.queryExecution.executedPlan)
--- End diff --
Sure! No problem at all :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58254140
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -81,29 +100,11 @@ package object debug {
* WholeStageCodegen subtree).
*/
def debugCodegen(): Unit = {
- debugPrint(debugCodegenString())
+ debugPrint(codegenString(query.queryExecution.executedPlan))
}
/** Visible for testing. */
-def debugCodegenString(): String = {
- val plan = query.queryExecution.executedPlan
- val codegenSubtrees = new
collection.mutable.HashSet[WholeStageCodegen]()
- plan transform {
-case s: WholeStageCodegen =>
- codegenSubtrees += s
- s
-case s => s
- }
- var output = s"Found ${codegenSubtrees.size} WholeStageCodegen
subtrees.\n"
- for ((s, i) <- codegenSubtrees.toSeq.zipWithIndex) {
-output += s"== Subtree ${i + 1} / ${codegenSubtrees.size} ==\n"
-output += s
-output += "\nGenerated code:\n"
-val (_, source) = s.doCodeGen()
-output += s"${CodeFormatter.format(source)}\n"
- }
- output
-}
+def debugCodegenString(): String =
codegenString(query.queryExecution.executedPlan)
--- End diff --
yea let's change that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58243529
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
Oh, do you mean `codegen select ...` instead `explain codegen select ...`?
I thought `codegen` was an boolean option flag, but new command is also
possible if it need more other options.
What kind of usage for `codegen` command do you think? Could you enumerate
some for me?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58243106
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -81,29 +100,11 @@ package object debug {
* WholeStageCodegen subtree).
*/
def debugCodegen(): Unit = {
- debugPrint(debugCodegenString())
+ debugPrint(codegenString(query.queryExecution.executedPlan))
}
/** Visible for testing. */
-def debugCodegenString(): String = {
- val plan = query.queryExecution.executedPlan
- val codegenSubtrees = new
collection.mutable.HashSet[WholeStageCodegen]()
- plan transform {
-case s: WholeStageCodegen =>
- codegenSubtrees += s
- s
-case s => s
- }
- var output = s"Found ${codegenSubtrees.size} WholeStageCodegen
subtrees.\n"
- for ((s, i) <- codegenSubtrees.toSeq.zipWithIndex) {
-output += s"== Subtree ${i + 1} / ${codegenSubtrees.size} ==\n"
-output += s
-output += "\nGenerated code:\n"
-val (_, source) = s.doCodeGen()
-output += s"${CodeFormatter.format(source)}\n"
- }
- output
-}
+def debugCodegenString(): String =
codegenString(query.queryExecution.executedPlan)
--- End diff --
It is still used in `DebuggingSuite` since I didn't change that testcase.
```
val res = sqlContext.range(10).groupBy("id").count().debugCodegenString()
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58166528
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala
---
@@ -101,4 +101,19 @@ class HiveExplainSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
"Physical Plan should not contain Subquery since it's eliminated
by optimizer")
}
}
+
+ test("SPARK-14251: EXPLAIN CODEGEN command") {
--- End diff --
Ok.
Regarding the name of the test. I would typically only add the JIRA number
to test name if it fixes a bug (that needs some explanation).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/12099#discussion_r58166375 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -576,7 +576,7 @@ frameBound explainOption -: LOGICAL | FORMATTED | EXTENDED +: LOGICAL | FORMATTED | EXTENDED | CODEGEN --- End diff -- Yeah it is fixed. He added it to the nonReserved rule (see below). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58165643
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
---
@@ -237,15 +237,18 @@ case class ExplainCommand(
logicalPlan: LogicalPlan,
override val output: Seq[Attribute] =
Seq(AttributeReference("plan", StringType, nullable = true)()),
-extended: Boolean = false)
+extended: Boolean = false,
+codegen: Boolean = false)
--- End diff --
maybe we should have a separate command? otherwise we are going to add a
lot of parameters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12099#discussion_r58165608 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -576,7 +576,7 @@ frameBound explainOption -: LOGICAL | FORMATTED | EXTENDED +: LOGICAL | FORMATTED | EXTENDED | CODEGEN --- End diff -- @hvanhovell does this fix the problem already? i wasn't sure what was before your comment vs after. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58165593
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala
---
@@ -101,4 +101,19 @@ class HiveExplainSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
"Physical Plan should not contain Subquery since it's eliminated
by optimizer")
}
}
+
+ test("SPARK-14251: EXPLAIN CODEGEN command") {
--- End diff --
it's really long -- i think having the first couple lines are good enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58165570
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -81,29 +100,11 @@ package object debug {
* WholeStageCodegen subtree).
*/
def debugCodegen(): Unit = {
- debugPrint(debugCodegenString())
+ debugPrint(codegenString(query.queryExecution.executedPlan))
}
/** Visible for testing. */
-def debugCodegenString(): String = {
- val plan = query.queryExecution.executedPlan
- val codegenSubtrees = new
collection.mutable.HashSet[WholeStageCodegen]()
- plan transform {
-case s: WholeStageCodegen =>
- codegenSubtrees += s
- s
-case s => s
- }
- var output = s"Found ${codegenSubtrees.size} WholeStageCodegen
subtrees.\n"
- for ((s, i) <- codegenSubtrees.toSeq.zipWithIndex) {
-output += s"== Subtree ${i + 1} / ${codegenSubtrees.size} ==\n"
-output += s
-output += "\nGenerated code:\n"
-val (_, source) = s.doCodeGen()
-output += s"${CodeFormatter.format(source)}\n"
- }
- output
-}
+def debugCodegenString(): String =
codegenString(query.queryExecution.executedPlan)
--- End diff --
do we still need this function?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204193533 Hi, @rxin . Could you review this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204191621 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54673/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204191619 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204191093 **[Test build #54673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54673/consoleFull)** for PR 12099 at commit [`83c3c01`](https://github.com/apache/spark/commit/83c3c01f8c938fb5247045d98e187294dabfcf8f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204181437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54669/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204181436 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204181205 **[Test build #54669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54669/consoleFull)** for PR 12099 at commit [`4e750e5`](https://github.com/apache/spark/commit/4e750e5cfecc8b1ac70c788c5811bc947d8acccb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204160580 **[Test build #54673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54673/consoleFull)** for PR 12099 at commit [`83c3c01`](https://github.com/apache/spark/commit/83c3c01f8c938fb5247045d98e187294dabfcf8f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204160591 Hi, @hvanhovell . I updated the PR according to your comments. For the real generated code, I think it's safe to maintain the first three lines since that will not change frequently. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204154576 Thank you, @hvanhovell . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user hvanhovell commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204154130 @dongjoon-hyun I left a few small comments. Looks pretty solid overall. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58136850
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala
---
@@ -101,4 +101,19 @@ class HiveExplainSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
"Physical Plan should not contain Subquery since it's eliminated
by optimizer")
}
}
+
+ test("SPARK-14251: EXPLAIN CODEGEN command") {
--- End diff --
Is it feasible to add an actual codegenerated example? Or do you think this
is too long/prone to changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12099#discussion_r58136464
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -122,9 +122,10 @@ class SparkSqlAstBuilder extends AstBuilder {
// Create the explain comment.
val statement = plan(ctx.statement)
if (isExplainableStatement(statement)) {
- ExplainCommand(statement, extended = options.exists(_.EXTENDED !=
null))
+ ExplainCommand(statement, extended = options.exists(_.EXTENDED !=
null),
+codegen = options.exists(_.CODEGEN != null))
} else {
- ExplainCommand(OneRowRelation)
+ ExplainCommand(OneRowRelation, codegen = options.exists(_.CODEGEN !=
null))
--- End diff --
is there a point showing the generated code of an otherwise unexplainable
command?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/12099#discussion_r58136340 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -576,7 +576,7 @@ frameBound explainOption -: LOGICAL | FORMATTED | EXTENDED +: LOGICAL | FORMATTED | EXTENDED | CODEGEN --- End diff -- `CODEGEN` is not a keyword. Please add to the `nonReserved` rule. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12099#issuecomment-204151657 **[Test build #54669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54669/consoleFull)** for PR 12099 at commit [`4e750e5`](https://github.com/apache/spark/commit/4e750e5cfecc8b1ac70c788c5811bc947d8acccb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14251][SQL] Add SQL command for printin...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/12099
[SPARK-14251][SQL] Add SQL command for printing out generated code for
debugging
## What changes were proposed in this pull request?
This PR implements `EXPLAIN CODEGEN` SQL command. In short, this command do
the `explain` and `debugCodegen` at one command. For example,
* (1) `sql("explain extended select 'a' as a group by
1").collect().foreach(println)`
* (2) `sql("select 'a' as a group by 1").debugCodegen()`
* (1+2: This PR) `sql("explain extended codegen select 'a' as a group by
1").collect().foreach(println)`
**Before**
```
scala> import org.apache.spark.sql.execution.debug._
scala> sql("explain extended select 'a' as a group by
1").collect().foreach(println)
[== Parsed Logical Plan ==]
...
[== Analyzed Logical Plan ==]
...
[== Optimized Logical Plan ==]
...
[== Physical Plan ==]
...
scala> sql("select 'a' as a group by 1").debugCodegen()
Found 2 WholeStageCodegen subtrees.
== Subtree 1 / 2 ==
...
Generated code:
...
== Subtree 2 / 2 ==
...
Generated code:
...
```
**After**
```
scala> sql("explain extended codegen select 'a' as a group by
1").collect().foreach(println)
[== Parsed Logical Plan ==]
...
[== Analyzed Logical Plan ==]
...
[== Optimized Logical Plan ==]
...
[== Physical Plan ==]
...
[]
[Found 2 WholeStageCodegen subtrees.]
[== Subtree 1 / 2 ==]
...
[]
[Generated code:]
...
[]
[== Subtree 2 / 2 ==]
...
[]
[Generated code:]
...
```
## How was this patch tested?
Pass the Jenkins tests (including new testcases)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-14251
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12099.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12099
commit 4e750e5cfecc8b1ac70c788c5811bc947d8acccb
Author: Dongjoon Hyun
Date: 2016-03-31T18:33:30Z
[SPARK-14251][SQL] Add SQL command for printing out generated code for
debugging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
