[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

2018-09-13 Thread gatorsmile 
Github user gatorsmile closed the pull request at:

https://github.com/apache/spark/pull/22403


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

2018-09-12 Thread gatorsmile 
Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/22403#discussion_r217265214
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
 ---
@@ -37,6 +38,7 @@ class BooleanSimplificationSuite extends PlanTest with 
PredicateHelper {
   Batch("Constant Folding", FixedPoint(50),
 NullPropagation(conf),
 ConstantFolding,
+SimplifyConditionals,
 BooleanSimplification,
 PruneFilters(conf)) :: Nil
   }
--- End diff --

I see. Thanks!



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

2018-09-12 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22403#discussion_r217171729
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
 ---
@@ -37,6 +38,7 @@ class BooleanSimplificationSuite extends PlanTest with 
PredicateHelper {
   Batch("Constant Folding", FixedPoint(50),
 NullPropagation(conf),
 ConstantFolding,
+SimplifyConditionals,
 BooleanSimplification,
 PruneFilters(conf)) :: Nil
   }
--- End diff --

At line 46, SPARK-17851 changed like the following. We need that.
```scala
- val testRelation = LocalRelation('a.int, 'b.int, 'c.int, 'd.string)
+ val testRelation = LocalRelation('a.int, 'b.int, 'c.int, 'd.string,
   'e.boolean, 'f.boolean, 'g.boolean, 'h.boolean)
```



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

2018-09-12 Thread gatorsmile 
GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/22403

[SPARK-25402][SQL][BACKPORT-2.2] Null handling in BooleanSimplification

## What changes were proposed in this pull request?
This PR is to fix the null handling in BooleanSimplification. In the rule 
BooleanSimplification, there are two cases that do not properly handle null 
values. The optimization is not right if either side is null. This PR is to fix 
them.

## How was this patch tested?
Added test cases


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark backportSpark25402

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22403


commit 7088371330591b5278ae4e6083cfcbedc5f0fda6
Author: gatorsmile 
Date:   2018-09-12T13:11:22Z

[SPARK-25402][SQL] Null handling in BooleanSimplification

This PR is to fix the null handling in BooleanSimplification. In the rule 
BooleanSimplification, there are two cases that do not properly handle null 
values. The optimization is not right if either side is null. This PR is to fix 
them.

Added test cases

Closes #22390 from gatorsmile/fixBooleanSimplification.

Authored-by: gatorsmile 
Signed-off-by: Wenchen Fan 
(cherry picked from commit 79cc59718fdf7785bdc37a26bb8df4c6151114a6)
Signed-off-by: Wenchen Fan 




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org