Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215620961
Right. Thank you so much for enriching ideas!
I'll update this PR with `FoldablePropagation`.
---
If your project is set up for it, you can reply to this emai
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215620606
the scope of `NullPropagation` is one operator, but we need a
`FoldablePropagation` whose scope is the whole plan tree. Think about `Sort(a,
Filter(true, Project(1 AS
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215620522
Oh, I got. Thanks. I will try to generalize.
* Sort(_, Project(_))
* Project(_, Project(...))
And so on.
---
If your project is set up for it, you ca
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215620369
If that's just about how to handle `Sort(_, Project(_,_))` expressions in
`EliminateSorts`, I can easily modify this PR according to your advice.
After moving
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215619413
Actually, `Sort` is dead end, we can not propagate up anymore.
So, in that case, removing looks more efficient.
Do you mean more generalized `Foldable
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215618718
`select 1 as a from tbl order by a` is equal to `select 1 as a from tbl
order by 1`.
When the child operator is `Project` and has foldable output, if the pare
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215616875
Thank you for review, @cloud-fan !
Do you mean removing aliases by replacing the base expression(?) by using
`transformUp`?
Maybe, except the top most alia
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215614987
instead of doing this, can we propagate foldable alias bottom up? So that
it's not only `Sort`, but all operators can benefit from it.
---
If your project is set up
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215564542
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215564535
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215564274
**[Test build #57271 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57271/consoleFull)**
for PR 12719 at commit
[`b7deb89`](https://g
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215547799
Thank you so much, @gatorsmile !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project d
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215546800
Sure, will do it tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215545987
Hi, @gatorsmile and @cloud-fan .
Could you take a look at this PR when you have some time?
I think you are the best persons to review this PR.
---
If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12719#issuecomment-215541446
**[Test build #57271 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57271/consoleFull)**
for PR 12719 at commit
[`b7deb89`](https://gi
15 matches
Mail list logo
