szehon-ho opened a new pull request, #57439: URL: https://github.com/apache/spark/pull/57439
### What changes were proposed in this pull request? This PR allows DataSource V2 connectors that consume Catalyst filters to opt in to partition-filter inference for identity-partitioned generated columns. When a query filters a generated column's base column, Spark derives a conservative filter on the generated partition column and includes it in filter pushdown. The supported generation expressions include casts to date, date/time extraction and truncation, `date_format`, `substring`, identity expressions, and composite year/month/day/hour partitions. Derived filters preserve null and truncation boundaries so they cannot remove matching rows. The PR also adds an in-memory test connector that evaluates Catalyst partition filters, unit coverage for expression recognition and derivation, and end-to-end coverage for pushdown and partition pruning. ### Why are the changes needed? Filters on a base column do not directly reference its generated partition column. Without inference, V2 data sources cannot use those filters for partition pruning and may scan every partition even when the generation expression makes a safe, selective partition predicate available. ### Does this PR introduce _any_ user-facing change? Yes. DataSource V2 implementations that explicitly opt in can receive additional generated-column partition filters and prune partitions more effectively. Query results are unchanged because every derived filter is implied by the original data filter and is used only for pushdown. ### How was this patch tested? Added `GeneratedColumnPartitionFilterSuite` for expression-level derivation and negative cases, and `DataSourceV2GeneratedColumnPartitionFilterSuite` for end-to-end pushdown, pruning, opt-out, null, parser-policy, case-sensitivity, nested-column, and multi-column behavior. Targeted test command (running at PR creation time): ```bash JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 \ MAVEN_MIRROR_URL=https://maven-proxy.cloud.databricks.com \ build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.v2.GeneratedColumnPartitionFilterSuite org.apache.spark.sql.execution.datasources.v2.DataSourceV2GeneratedColumnPartitionFilterSuite' ``` Also verified `git diff --check`, non-ASCII content, changed-file line lengths, and IDE diagnostics. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor (GPT-5.6 Sol) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
