Hi all, I would like to propose enhancements for partition filter pushdown, for DSV2 data sources that support partitioning (ie, those with partition stats).
Some DSV2 data sources, for example table formats like Apache Iceberg, lack partition filtering in many queries, compared to Spark-native data sources that directly use Catalyst (like Parquet). This proposal can bridge that gap while simplifying the data source logic. JIRA: https://issues.apache.org/jira/browse/SPARK-55596 SPIP doc: https://docs.google.com/document/d/17vcw411PxSRLWoK-BiLI56UiNdokLWtovF8JZUlDTOo Look forward to comments and feedback. Thanks, Szehon
