Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
@hvanhovell OK. Let's see if we can have a proper CNF soon. Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14912
@viirya TBH this seems hacky to me and I'd rather not merge this. I think
we should just focus on having proper CNF in the optimizer. I am sorry to
disappoint you.
---
If your project is set
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
ping @cloud-fan @hvanhovell @srinathshankar Can you take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
ping @cloud-fan @hvanhovell @srinathshankar again, please take look if you
have time. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
ping @cloud-fan @hvanhovell Can you review this if you have time? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
ping @cloud-fan @hvanhovell @srinathshankar again, would you please take a
look this? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user nsyca commented on the issue:
https://github.com/apache/spark/pull/14912
@viirya, I agree that we need a separate set of PRs to address the general
problem.
On your comment: "I think the goal to simplify a predicate such as (a > 10
|| b > 2) && (a > 10 || c == 3)
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14912
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14912
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65298/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14912
**[Test build #65298 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65298/consoleFull)**
for PR 14912 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14912
**[Test build #65298 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65298/consoleFull)**
for PR 14912 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
ping @srinathshankar @cloud-fan @hvanhovell Can you help review this change?
Some context here:
Some predicates are unable to push down because:
1. Predicates are simplified
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
@nsyca Thanks for your detailed comment. I would like to leave the decision
of predicate transformation to later PRs, as this PR is not motivated by this.
I think to simplify a predicate
Github user nsyca commented on the issue:
https://github.com/apache/spark/pull/14912
Thanks, @gatorsmile, for mentioning me. I will try my best to comment on
this thread. Disclaimer: I have not looked at the existing code manipulating
predicates/expressions in Spark. Nor have I the
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
To maintain the predicate sets may increase much complexity as I can think.
I don't know how big the set could be. But once you change one of the
predicates, you need to construct all equivalent
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14912
I am thinking whether it makes more sense to maintain multiple semantically
equivalent predicate sets for each `Filter`. In your example, we have both `(a
> 10 || b > 2) && (a > 10 || c == 3)`
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
also cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
@gatorsmile I've described it in the pr description.
Simply said, now a Filter will be stopped to pushdown once it encounters
another Filter. `BooleanSimplification` rule will simplify the
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14912
Could you define the conditions in which the predicates are unable to be
pushed down? Then, we can easily justify the significance.
---
If your project is set up for it, you can reply to this
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
@srinathshankar @gatorsmile I think CNF is another issue other then the
issue this PR was proposed to solve at the first. I would like to solve the
original adjoining Filter pushdown problem here.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
The CNF exponential expansion issue is an important concern in previous
works. Actually you can find that this patch doesn't produce a real CNF for
predicate. I use `splitDisjunctivePredicates` to
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
hmm, looks like there are previous works regarding CNF but none of them are
really merged. @gatorsmile Thanks for the context.
---
If your project is set up for it, you can reply to this email and
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14912
@viirya Could you please wait for the CNF predicate normalization rule?
@liancheng @yjshen did a few related work before. See
https://github.com/apache/spark/pull/10444 and
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14912
@srinathshankar I've addressed your comments. Please take a look. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14912
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14912
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64929/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14912
**[Test build #64929 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64929/consoleFull)**
for PR 14912 at commit
27 matches
Mail list logo
