pan3793 opened a new pull request, #58702: URL: https://github.com/apache/spark/pull/58702
### What changes were proposed in this pull request? `PushDownUtils.getPartitionPredicateSchema` returned a schema only when every transform in `Table.partitioning()` is an identity transform. It now accepts any partitioning with at least one identity transform: identity fields can be referenced by a `PartitionPredicate`, other fields keep their ordinal but are never referenced, so a filter on the source column of a non-identity transform stays a data filter. The connector contract is unchanged: `PartitionPredicate.eval` still receives the full partition key, and `references()` still reports ordinals into `Table.partitioning()`. This applies to all three users of the schema: the static second pass, runtime filter pushdown (DPP and scalar subqueries), and the metadata-only DELETE rewrite. The in-memory V2 filter test table now accepts only predicates of the shape it can evaluate, a column against a literal, and returns anything else, e.g. a predicate over a cast, as a real connector would. ### Why are the changes needed? A table partitioned by, for example, `dt STRING` (identity) and `bucket(16, user_id)` received no `PartitionPredicate` at all. A filter such as `dt = DATE'2026-09-01'` is analyzed as `cast(dt AS DATE) = DATE'2026-09-01'`; a connector that does not evaluate casts returns it from the first pass (and without ANSI mode it is not translatable at all), so only a `PartitionPredicate` can prune with it. On such a table it could not prune partitions in either the static or the runtime path, and could not drive a metadata-only DELETE, while the same filter on an identity-only table can. Mixed partitionings are the common case for connectors that support partition transforms. The all-identity check was raised in the SPARK-55596 review (#54459) and kept only because the partition-key contract was still open then. The contract that shipped (full key, partitioning order) is what makes lifting the check safe. ### Does this PR introduce _any_ user-facing change? No. A connector that opts into iterative pushdown and has a mixed partitioning now receives `PartitionPredicate`s over its identity fields; the `eval` contract is unchanged. ### How was this patch tested? New tests on a partitioning of one identity column plus a bucket transform: a predicate on the identity field is pushed and prunes in the static pass, via DPP, and in a metadata-only DELETE, including the `dt STRING` vs `DATE` literal case whose type-coercion cast the source cannot evaluate; an identity field placed after the bucket transform binds to its own ordinal; a filter on the bucket source column is not turned into a predicate; a bucket-only partitioning yields no predicate; the predicate survives Java and Kryo round-trips. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
