[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user liancheng closed the pull request at: https://github.com/apache/spark/pull/11200 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-184019253 Oh, I didn't realize that those extra vertical lines are used to indicate scopes of codegen'd stages, then the original tree string makes sense. I thought they were dangling connecting lines. Before whole stage codege: ``` Union :- Project [id#3L AS a#6L] : +- Range 0, 1, 8, 5, [id#3L] +- Project [id#4L AS a#7L] +- Range 0, 1, 8, 2, [id#4L] ``` After whole stage codege (before this PR): ``` Union :- WholeStageCodegen : : +- Project [id#3L AS a#6L] : : +- Range 0, 1, 8, 5, [id#3L] +- WholeStageCodegen : +- Project [id#4L AS a#7L] : +- Range 0, 1, 8, 2, [id#4L] ``` I'm closing this PR then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-184015792 I'm really confused. what were we doing before whole-stage codegen? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-184013932 At least there should be a vertical tree line connecting two `WholeStageCodegen` nodes to indicate that they are both children of `Union`? The `:` in plan tree string used to indicate that. Are you worrying about that users may mistake sub-trees under `WholeStageCodegen` for actual offspring nodes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-183968642 I think both are wrong. We want something like ``` == Physical Plan == Union :- WholeStageCodegen : +- Project [id#3L AS a#6L] : +- Range 0, 1, 8, 5, [id#3L] +- WholeStageCodegen : +- Project [id#4L AS a#7L] : +- Range 0, 1, 8, 2, [id#4L] ``` The ":" indicates the scope of whole stage codegen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-183898723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51267/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-183898257 **[Test build #51267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51267/consoleFull)** for PR 11200 at commit [`79a4cb4`](https://github.com/apache/spark/commit/79a4cb436b076fdab842cf3a441a627a5c1de729). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-183898718 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/11200
[SPARK-13314][SQL] Fixes malformed WholeStageCodegen tree string
When the query plan contains binary operator(s), `WholeStageCodegen` tree
string can be malformed. This PR fixes this issue.
Before:
```
val a = sqlContext range 5
val b = sqlContext range 2
a select ('id as 'a) unionAll (b select ('id as 'a)) explain true
...
== Physical Plan ==
Union
:- WholeStageCodegen
: : +- Project [id#3L AS a#6L]
: : +- Range 0, 1, 8, 5, [id#3L]
+- WholeStageCodegen
: +- Project [id#4L AS a#7L]
: +- Range 0, 1, 8, 2, [id#4L]
```
After:
```
== Physical Plan ==
Union
:- WholeStageCodegen
: +- Project [id#0L AS a#2L]
:+- Range 0, 1, 8, 5, [id#0L]
+- WholeStageCodegen
+- Project [id#1L AS a#3L]
+- Range 0, 1, 8, 2, [id#1L]
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark
spark-13314-fix-malformed-tree-string
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11200.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11200
commit 79a4cb436b076fdab842cf3a441a627a5c1de729
Author: Cheng Lian
Date: 2016-02-14T12:36:35Z
Fixes malformed WholeStageCodegen tree string
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-183882394 cc @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13314][SQL] Fixes malformed WholeStageC...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11200#issuecomment-183883876 **[Test build #51267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51267/consoleFull)** for PR 11200 at commit [`79a4cb4`](https://github.com/apache/spark/commit/79a4cb436b076fdab842cf3a441a627a5c1de729). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
