GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22368
[SPARK-25368][SQL] Incorrect predicate pushdown returns wrong result ## What changes were proposed in this pull request? How to reproduce: ```scala val df1 = spark.createDataFrame(Seq( (1, 1) )).toDF("a", "b").withColumn("c", lit(null).cast("int")) val df2 = df1.union(df1).withColumn("d", spark_partition_id).filter($"c".isNotNull) df2.show +---+---+----+---+ | a| b| c| d| +---+---+----+---+ | 1| 1|null| 0| | 1| 1|null| 1| +---+---+----+---+ ``` `filter($"c".isNotNull)`changed to `(null <=> c#10)` before https://github.com/apache/spark/pull/19201, but it changed to `(c#10 = null)` since https://github.com/apache/spark/pull/20155. This pr revert it to `(null <=> c#10)` to fix this issue. ## How was this patch tested? unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangyum/spark SPARK-25368 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22368.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22368 ---- commit 86b9b7892c94be68145453f9519e35a3574fe568 Author: Yuming Wang <yumwang@...> Date: 2018-09-09T03:46:18Z Fix SPARK-25368 commit 865e0af572edad7fd775c25e317055ffa0df2a08 Author: Yuming Wang <yumwang@...> Date: 2018-09-09T04:22:29Z Fix InferFiltersFromConstraintsSuite test error ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org