Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/20541
@gatorsmileï¼ OKï¼ I will do it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/20541
oh, I see, I fallback to the modification of the non-deterministic
expression, and to keep the newly added test cases for a+1 and a+b, can you?
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20541
I don't agree. `a + 1`/`a + b` are evaluated the same number of time, no
matter you push in through Union or not. I don't see any performance benefit by
doing this, except you can eliminate the
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/20541
oh ,yeah, there is a little difference, for a + 1 and a + b.
**for a + 1**:
```
`PushProjectionThroughUnion `rule handles:
Union
:- Project [(a#0 + 1) AS aa#10]
: +-
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20541
`ColumnPruning` rule handles `Union` already.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/20541
in my opinion, this is considered that PushProjectionThroughUnion optimizes
rules when there are multiple columns of union in data sources, while
projection requires only a few columns, and the
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20541
I think the use case is, by pushing projects into Union, we are more likely
to combine adjacent Unions. So I don't think we need to improve it to push part
of the project list and still leave a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87237/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20541
**[Test build #87237 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87237/testReport)**
for PR 20541 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20541
I'm confused about why we need `PushProjectionThroughUnion`. Generally we
only need to push down required columns, not entire project list, as there is
no benifit of doing this. I think we just
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20541
**[Test build #87237 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87237/testReport)**
for PR 20541 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87210/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20541
**[Test build #87210 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87210/testReport)**
for PR 20541 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20541
**[Test build #87210 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87210/testReport)**
for PR 20541 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20541
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20541
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/20541
@gatorsmile ,@cloud-fan Can you help me to review it. thanks.
---
-
To unsubscribe, e-mail:
23 matches
Mail list logo
