[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222343017
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59576/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222343015
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222342993
  
**[Test build #59576 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59576/consoleFull)**
 for PR 13147 at commit 
[`fbc9a9f`](https://github.com/apache/spark/commit/fbc9a9f08b65e1210e9db9eb92a9769bcfb693a9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222341221
  
**[Test build #59576 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59576/consoleFull)**
 for PR 13147 at commit 
[`fbc9a9f`](https://github.com/apache/spark/commit/fbc9a9f08b65e1210e9db9eb92a9769bcfb693a9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222339354
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59573/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222339353
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222339329
  
**[Test build #59573 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59573/consoleFull)**
 for PR 13147 at commit 
[`7b3f4a1`](https://github.com/apache/spark/commit/7b3f4a1ac99a3e4ab2648288e3a0b71a7e2defd5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222337092
  
**[Test build #59573 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59573/consoleFull)**
 for PR 13147 at commit 
[`7b3f4a1`](https://github.com/apache/spark/commit/7b3f4a1ac99a3e4ab2648288e3a0b71a7e2defd5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-94642
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59553/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-94640
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-28 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-94606
  
**[Test build #59553 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59553/consoleFull)**
 for PR 13147 at commit 
[`e5ee2c2`](https://github.com/apache/spark/commit/e5ee2c2b4c7351df42d9096b95f4ed3759f540d7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class SparkPlannerSuite extends SharedSQLContext `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-91941
  
**[Test build #59553 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59553/consoleFull)**
 for PR 13147 at commit 
[`e5ee2c2`](https://github.com/apache/spark/commit/e5ee2c2b4c7351df42d9096b95f4ed3759f540d7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-85954
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59545/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-85952
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-85934
  
**[Test build #59545 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59545/consoleFull)**
 for PR 13147 at commit 
[`a3524cd`](https://github.com/apache/spark/commit/a3524cd220f840839a7f40c570fe2c5da4c7705b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread ueshin
Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/13147#discussion_r64982752
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
 ---
@@ -47,17 +55,27 @@ abstract class QueryPlanner[PhysicalPlan <: 
TreeNode[PhysicalPlan]] {
   /** A list of execution strategies that can be used by the planner */
   def strategies: Seq[GenericStrategy[PhysicalPlan]]
 
-  /**
-   * Returns a placeholder for a physical plan that executes `plan`. This 
placeholder will be
-   * filled in automatically by the QueryPlanner using the other execution 
strategies that are
-   * available.
-   */
-  protected def planLater(plan: LogicalPlan): PhysicalPlan = 
this.plan(plan).next()
-
   def plan(plan: LogicalPlan): Iterator[PhysicalPlan] = {
 // Obviously a lot to do here still...
-val iter = strategies.view.flatMap(_(plan)).toIterator
+val iter = strategies.view.flatMap(_(plan)).toIterator.flatMap { 
physicalPlan =>
+  val placeholders = collectPlaceholders(physicalPlan)
+
+  (Iterator(physicalPlan) /: placeholders.toIterator) {
+case (physicalPlans, (placeholder, logicalPlan)) =>
+  val children = this.plan(logicalPlan)
+  physicalPlans.flatMap { physicalPlan =>
+children.map { child =>
+  physicalPlan.transformUp {
+case `placeholder` => child
+  }
+}
+  }
+  }
+}
+// TODO: We will need to prune bad plans to prevent from combinatorial 
explosion.
--- End diff --

Ah, I'm sorry, you are right because the planner returns plans as Iterator 
so the any other branches should not be processed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-83201
  
**[Test build #59545 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59545/consoleFull)**
 for PR 13147 at commit 
[`a3524cd`](https://github.com/apache/spark/commit/a3524cd220f840839a7f40c570fe2c5da4c7705b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread ueshin
Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/13147#discussion_r64981719
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
 ---
@@ -47,17 +55,27 @@ abstract class QueryPlanner[PhysicalPlan <: 
TreeNode[PhysicalPlan]] {
   /** A list of execution strategies that can be used by the planner */
   def strategies: Seq[GenericStrategy[PhysicalPlan]]
 
-  /**
-   * Returns a placeholder for a physical plan that executes `plan`. This 
placeholder will be
-   * filled in automatically by the QueryPlanner using the other execution 
strategies that are
-   * available.
-   */
-  protected def planLater(plan: LogicalPlan): PhysicalPlan = 
this.plan(plan).next()
-
   def plan(plan: LogicalPlan): Iterator[PhysicalPlan] = {
 // Obviously a lot to do here still...
-val iter = strategies.view.flatMap(_(plan)).toIterator
+val iter = strategies.view.flatMap(_(plan)).toIterator.flatMap { 
physicalPlan =>
+  val placeholders = collectPlaceholders(physicalPlan)
+
+  (Iterator(physicalPlan) /: placeholders.toIterator) {
+case (physicalPlans, (placeholder, logicalPlan)) =>
+  val children = this.plan(logicalPlan)
+  physicalPlans.flatMap { physicalPlan =>
+children.map { child =>
+  physicalPlan.transformUp {
+case `placeholder` => child
+  }
+}
+  }
+  }
+}
+// TODO: We will need to prune bad plans to prevent from combinatorial 
explosion.
--- End diff --

We are not safe now if some strategy returns two or more plans.
Should we take the first plan here for now?
(I will introduce `prune` method and implement it in `SparkPlanner`.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread marmbrus
Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-63996
  
This is great!  In order to commit this it would be good to explain what is 
happening in detail (I think I follow, but its very dense).  I'd also like to 
see some tests to make sure that we are finding the first plan without going 
down any of the possible branches.

One consideration: we might want to make at least the API parts of this 
make it into 2.0 since it makes plugging in strategies much more powerful and 
it does change this (experimental) interface.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread marmbrus
Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/13147#discussion_r64972461
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
 ---
@@ -47,17 +55,27 @@ abstract class QueryPlanner[PhysicalPlan <: 
TreeNode[PhysicalPlan]] {
   /** A list of execution strategies that can be used by the planner */
   def strategies: Seq[GenericStrategy[PhysicalPlan]]
 
-  /**
-   * Returns a placeholder for a physical plan that executes `plan`. This 
placeholder will be
-   * filled in automatically by the QueryPlanner using the other execution 
strategies that are
-   * available.
-   */
-  protected def planLater(plan: LogicalPlan): PhysicalPlan = 
this.plan(plan).next()
-
   def plan(plan: LogicalPlan): Iterator[PhysicalPlan] = {
 // Obviously a lot to do here still...
-val iter = strategies.view.flatMap(_(plan)).toIterator
+val iter = strategies.view.flatMap(_(plan)).toIterator.flatMap { 
physicalPlan =>
+  val placeholders = collectPlaceholders(physicalPlan)
+
+  (Iterator(physicalPlan) /: placeholders.toIterator) {
+case (physicalPlans, (placeholder, logicalPlan)) =>
+  val children = this.plan(logicalPlan)
+  physicalPlans.flatMap { physicalPlan =>
+children.map { child =>
+  physicalPlan.transformUp {
+case `placeholder` => child
+  }
+}
+  }
+  }
+}
+// TODO: We will need to prune bad plans to prevent from combinatorial 
explosion.
--- End diff --

We are safe for now though as long as we only call `next()` once right?  Is 
there some way to test that its doing the right thing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread marmbrus
Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/13147#discussion_r64972400
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
 ---
@@ -47,17 +55,27 @@ abstract class QueryPlanner[PhysicalPlan <: 
TreeNode[PhysicalPlan]] {
   /** A list of execution strategies that can be used by the planner */
   def strategies: Seq[GenericStrategy[PhysicalPlan]]
 
-  /**
-   * Returns a placeholder for a physical plan that executes `plan`. This 
placeholder will be
-   * filled in automatically by the QueryPlanner using the other execution 
strategies that are
-   * available.
-   */
-  protected def planLater(plan: LogicalPlan): PhysicalPlan = 
this.plan(plan).next()
-
   def plan(plan: LogicalPlan): Iterator[PhysicalPlan] = {
 // Obviously a lot to do here still...
-val iter = strategies.view.flatMap(_(plan)).toIterator
+val iter = strategies.view.flatMap(_(plan)).toIterator.flatMap { 
physicalPlan =>
+  val placeholders = collectPlaceholders(physicalPlan)
+
+  (Iterator(physicalPlan) /: placeholders.toIterator) {
+case (physicalPlans, (placeholder, logicalPlan)) =>
+  val children = this.plan(logicalPlan)
+  physicalPlans.flatMap { physicalPlan =>
+children.map { child =>
+  physicalPlan.transformUp {
+case `placeholder` => child
--- End diff --

You might also explicitly spell out what is happening here.  It's doing the 
following right?

```scala
case p if p == placeholder => child
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread marmbrus
Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/13147#discussion_r64972060
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
 ---
@@ -47,17 +55,27 @@ abstract class QueryPlanner[PhysicalPlan <: 
TreeNode[PhysicalPlan]] {
   /** A list of execution strategies that can be used by the planner */
   def strategies: Seq[GenericStrategy[PhysicalPlan]]
 
-  /**
-   * Returns a placeholder for a physical plan that executes `plan`. This 
placeholder will be
-   * filled in automatically by the QueryPlanner using the other execution 
strategies that are
-   * available.
-   */
-  protected def planLater(plan: LogicalPlan): PhysicalPlan = 
this.plan(plan).next()
-
   def plan(plan: LogicalPlan): Iterator[PhysicalPlan] = {
 // Obviously a lot to do here still...
-val iter = strategies.view.flatMap(_(plan)).toIterator
+val iter = strategies.view.flatMap(_(plan)).toIterator.flatMap { 
physicalPlan =>
+  val placeholders = collectPlaceholders(physicalPlan)
+
+  (Iterator(physicalPlan) /: placeholders.toIterator) {
--- End diff --

Style: I'm pretty sure @rxin will cry if we use `/:` 😄 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222165314
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59491/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222165313
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222165013
  
**[Test build #59491 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59491/consoleFull)**
 for PR 13147 at commit 
[`42c18ab`](https://github.com/apache/spark/commit/42c18ab847ab8b1e112a3a3129bd7483b21a486c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222144767
  
**[Test build #59491 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59491/consoleFull)**
 for PR 13147 at commit 
[`42c18ab`](https://github.com/apache/spark/commit/42c18ab847ab8b1e112a3a3129bd7483b21a486c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread ueshin
Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222143950
  
@marmbrus Thank you for the design sketch.
I'd like you to see my approach I was thinking after my first commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222124875
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222124878
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59484/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222124696
  
**[Test build #59484 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59484/consoleFull)**
 for PR 13147 at commit 
[`0dccae8`](https://github.com/apache/spark/commit/0dccae8fe0c07d085880ea5feec6fafd9848dbc4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222110844
  
**[Test build #59484 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59484/consoleFull)**
 for PR 13147 at commit 
[`0dccae8`](https://github.com/apache/spark/commit/0dccae8fe0c07d085880ea5feec6fafd9848dbc4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222094657
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222094659
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59474/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222094433
  
**[Test build #59474 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59474/consoleFull)**
 for PR 13147 at commit 
[`85d412a`](https://github.com/apache/spark/commit/85d412aa2431333ecf0ba8de270777b0a994fcb3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `abstract class SparkStrategy extends GenericStrategy[SparkPlan] `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222078986
  
**[Test build #59474 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59474/consoleFull)**
 for PR 13147 at commit 
[`85d412a`](https://github.com/apache/spark/commit/85d412aa2431333ecf0ba8de270777b0a994fcb3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [WIP][SPARK-6320][SQL] Move planLater method i...

2016-05-27 Thread ueshin
Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/13147#issuecomment-222078722
  
This is a simple wip version with `PlanLater` placeholder.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org