Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-197724073
@gatorsmile @rxin @cloud-fan Since this PR does not solve the problem as
expected, also introduce other problems (can't prune columns for
Filter(Join(xx)), I have revert
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-197725609
@davies Sorry for that.
Could you review another PR: https://github.com/apache/spark/pull/11745 ?
That is built on this PR to resolve all the issues. Thanks
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11682#discussion_r56459598
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -410,6 +401,24 @@ object ColumnPruning extends Rule[Logi
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11682#discussion_r56459726
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -410,6 +401,24 @@ object ColumnPruning extends Rule[
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11682#discussion_r56459774
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -410,6 +401,24 @@ object ColumnPruning extends Rule[
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-197004091
After writing more test cases, I found a couple of issues. In addition, we
need another split of the rule `ColumnPruning`. Have a new rule
`PushProjectThroughPredica
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196939816
@davies I might understand your points. We still prefer
`PushPredicateThroughProject`. Let me submit another PR to address that issue
and at you.
Let me ex
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196922074
@gatorsmile This latest changes does not address the problem in the PR
description, ColumnPruning and PushPredicateThroughProject still conflict with
each other, right?
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196839488
@davies In the case Filter(Project()), the second rule `EliminateOperators`
removes the useless Project if it does not prune any column.
Could you explain a
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196732337
@gatorsmile Have you missed the special rule for Filter(Project()) ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11682
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196701013
Merging in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196628698
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196628692
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196627725
**[Test build #53144 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53144/consoleFull)**
for PR 11682 at commit
[`bc4685a`](https://g
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196617839
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enable
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11682#issuecomment-196594548
**[Test build #53144 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53144/consoleFull)**
for PR 11682 at commit
[`bc4685a`](https://gi
17 matches
Mail list logo
