[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/17660 @cloud-fan switching to lazy vals to avoid these predicates being evaluated when they are not used seems to work. so i think this is a better (more targeted) solution for now, and i removed

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17660 **[Test build #75902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75902/testReport)** for PR 17660 at commit

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/17660 I see. let me check if making leftHasNonNullPredicate and rightHasNonNullPredicate lazy solves it then On Apr 17, 2017 23:44, "Wenchen Fan" wrote:

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17660 In general, we can't decide whether a UDF is null-propagate or not, so we can't catch NPE for `Expression.eval` blindly. When we call `Expression.eval` in optimizer, we should make sure we do not

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17660 I think the root problem is, in `EliminateOuterJoin.buildNewJoinType`, we always build `leftHasNonNullPredicate` and `rightHasNonNullPredicate`. If it's left join, only `rightHasNonNullPredicate`

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17660 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75863/ Test PASSed. ---

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17660 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17660 **[Test build #75863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75863/testReport)** for PR 17660 at commit

[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17660 **[Test build #75863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75863/testReport)** for PR 17660 at commit