Jackey Lee created SPARK-38427:
----------------------------------
Summary: DataFilter pushed down with PartitionFilter for Orc
Key: SPARK-38427
URL: https://issues.apache.org/jira/browse/SPARK-38427
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.3.0
Reporter: Jackey Lee
At present, for orc data source, the Filter is divided into DataFilter and
PartitionFilter when it is pushed down, but when the Filter removes the
PartitionFilter, it means that all Partitions will scan all DataFilter
conditions, which may cause full data scan.
Based on SPARK-38041, we can pushdown dataFilter with partitionFilter to ORC,
and remove partitionFilter at runtime.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
