Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194694790
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194694793
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194694601
**[Test build #52815 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52815/consoleFull)**
for PR 11565 at commit
[`4dd3e66`](https://g
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55637701
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55636439
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55636068
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55634864
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -387,6 +381,14 @@ object ColumnPruning extends Rule[L
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194669064
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194668856
**[Test build #52790 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52790/consoleFull)**
for PR 11565 at commit
[`b229ea2`](https://g
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194669060
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194664648
**[Test build #52815 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52815/consoleFull)**
for PR 11565 at commit
[`4dd3e66`](https://gi
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55633862
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -326,9 +326,9 @@ object ColumnPruning extends Rule[L
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55633324
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -326,9 +326,9 @@ object ColumnPruning extends Rule[L
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194654478
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194654480
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194654313
**[Test build #52806 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52806/consoleFull)**
for PR 11565 at commit
[`f0fbe78`](https://g
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55630583
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -326,9 +326,9 @@ object ColumnPruning extends Rule[Lo
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194635702
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194635704
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194635465
**[Test build #52796 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52796/consoleFull)**
for PR 11565 at commit
[`60fcafa`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194625407
**[Test build #52806 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52806/consoleFull)**
for PR 11565 at commit
[`f0fbe78`](https://gi
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194621904
A good idea! but I need to change it to
```(AttributeSet(w.windowExpressions.map(_.toAttribute)) --
p.references).nonEmpty```.
If `Project` will n
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194617786
how about we change `(w.outputSet -- p.references).nonEmpty` to
`AttributeSet(w.windowExpressions -- p.references).nonEmpty`?
---
If your project is set up for it, y
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194601345
**[Test build #52796 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52796/consoleFull)**
for PR 11565 at commit
[`60fcafa`](https://gi
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194600791
@cloud-fan Your concern is valid.
When processing the child Window, it will always enter the case:
https://github.com/apache/spark/blob/master/sql/catal
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-194596383
**[Test build #52790 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52790/consoleFull)**
for PR 11565 at commit
[`b229ea2`](https://gi
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193630940
Let me add the test cases in column pruning through `Windows` in
`ColumnPruningSuite` tonight/tomorrow. Thanks!
---
If your project is set up for it, you can reply
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193628390
ah I see, makes sense. It would be better if there are some tests which
guarantee the column pruning for window still works. Otherwise LGTM.
cc @yhuai
---
If yo
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193616509
We still can prune the child even if we do not have `projectList`. See the
following code:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scal
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193614610
After think about it more, I think the `projectList` maybe useful to do
column pruning for `Window`.
If there is a `Project` above `Window`, it's easy to filt
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55317197
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -345,8 +343,6 @@ object ColumnPruning extends Rule[Lo
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55314448
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -345,8 +343,6 @@ object ColumnPruning extends Rule[L
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193583045
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193583046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193582844
**[Test build #52620 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52620/consoleFull)**
for PR 11565 at commit
[`467b095`](https://g
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55311096
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -345,8 +343,6 @@ object ColumnPruning extends Rule[Logic
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193550327
**[Test build #52620 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52620/consoleFull)**
for PR 11565 at commit
[`467b095`](https://gi
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55304511
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -345,8 +343,6 @@ object ColumnPruning extends Rule[L
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11565#discussion_r55301619
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -345,8 +343,6 @@ object ColumnPruning extends Rule[Lo
Github user frreiss commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193533582
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193526639
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193526633
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193526313
**[Test build #52602 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52602/consoleFull)**
for PR 11565 at commit
[`25f6ff6`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11565#issuecomment-193487512
**[Test build #52602 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52602/consoleFull)**
for PR 11565 at commit
[`25f6ff6`](https://gi
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/11565
[SPARK-13732] [SQL] Remove projectList from Window
What changes were proposed in this pull request?
`projectList` is useless. Its value is always the same as the child.output.
Remo
45 matches
Mail list logo
