[ https://issues.apache.org/jira/browse/SPARK-24934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562126#comment-16562126 ]
Hyukjin Kwon commented on SPARK-24934: -------------------------------------- I think this has been a bug from the first place. It at least affects 2.3.1. I manually tested: {code} Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.1 /_/ Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162) Type in expressions to have them evaluated. Type :help for more information. scala> import org.apache.spark.sql.functions import org.apache.spark.sql.functions scala> scala> val df = Seq(Array("a", "b"), Array("c", "d")).toDF("arrayCol") df: org.apache.spark.sql.DataFrame = [arrayCol: array<string>] scala> df.filter(df.col("arrayCol").eqNullSafe(functions.array(functions.lit("a"), functions.lit("b")))).show() +--------+ |arrayCol| +--------+ | [a, b]| +--------+ scala> df.cache().filter(df.col("arrayCol").eqNullSafe(functions.array(functions.lit("a"), functions.lit("b")))).show() +--------+ |arrayCol| +--------+ +--------+ {code} > Complex type and binary type in in-memory partition pruning does not work due > to missing upper/lower bounds cases > ----------------------------------------------------------------------------------------------------------------- > > Key: SPARK-24934 > URL: https://issues.apache.org/jira/browse/SPARK-24934 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.1, 2.4.0 > Reporter: Hyukjin Kwon > Assignee: Hyukjin Kwon > Priority: Critical > Labels: correctness > Fix For: 2.3.2, 2.4.0 > > > For example, if array is used (where the lower and upper bounds for its > column batch are {{null}})), it looks wrongly filtering all data out: > {code} > scala> import org.apache.spark.sql.functions > import org.apache.spark.sql.functions > scala> val df = Seq(Array("a", "b"), Array("c", "d")).toDF("arrayCol") > df: org.apache.spark.sql.DataFrame = [arrayCol: array<string>] > scala> > df.filter(df.col("arrayCol").eqNullSafe(functions.array(functions.lit("a"), > functions.lit("b")))).show() > +--------+ > |arrayCol| > +--------+ > | [a, b]| > +--------+ > scala> > df.cache().filter(df.col("arrayCol").eqNullSafe(functions.array(functions.lit("a"), > functions.lit("b")))).show() > +--------+ > |arrayCol| > +--------+ > +--------+ > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org