[ https://issues.apache.org/jira/browse/SPARK-32352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182798#comment-17182798 ]
Apache Spark commented on SPARK-32352: -------------------------------------- User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/29526 > Partially push down support data filter if it mixed in partition filters > ------------------------------------------------------------------------- > > Key: SPARK-32352 > URL: https://issues.apache.org/jira/browse/SPARK-32352 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Reporter: Yuming Wang > Assignee: angerszhu > Priority: Major > Fix For: 3.1.0 > > > We support partially push partition filters since SPARK-28169. We can also > support partially push down data filters if it mixed in partition filters and > data filters. For example: > {code:scala} > spark.sql( > s""" > |CREATE TABLE t(i INT, p STRING) > |USING parquet > |PARTITIONED BY (p)""".stripMargin) > spark.range(0, 1000).selectExpr("id as col").createOrReplaceTempView("temp") > for (part <- Seq(1, 2, 3, 4)) { > sql(s""" > |INSERT OVERWRITE TABLE t PARTITION (p='$part') > |SELECT col FROM temp""".stripMargin) > } > spark.sql("SELECT * FROM t WHERE WHERE (p = '1' AND i = 1) OR (p = '2' and i > = 2)").explain() > {code} > We can also push down {{ i = 1 or i = 2 }}. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org