pan3793 opened a new pull request, #58851:
URL: https://github.com/apache/spark/pull/58851
### What changes were proposed in this pull request?
`PartitionPredicateImpl.eval` reported a partition as matching whenever it
could not evaluate
the predicate, either because evaluating the expression threw or because the
partition key width
did not match the schema. It now propagates that failure.
The old behavior is kept for runtime filters only, where Spark evaluates the
filter again in the
post-scan `FilterExec`, so failing open costs a pruning opportunity and
nothing else.
`createRuntimePartitionPredicates` is the only caller that asks for it.
This also documents the `eval` contract for connectors, and corrects a
comment in the in-memory
test table that stated the old justification as a general rule.
### Why are the changes needed?
Failing open is sound only while something else still evaluates the filter.
Two paths have not
met that condition since `PartitionPredicate` was introduced in 4.2.0:
- the static second pass, where an accepted `PartitionPredicate` is removed
from the post-scan
filters;
- the metadata-only DELETE rewrite, which has no post-scan filter at all.
On both, the predicate is Spark's only evaluator, so reporting a match on
failure is a wrong
answer. On a table partitioned by `dep STRING` holding `'hr'` and `'1'`,
with a filter on `dep`
whose evaluation fails for `'hr'`, an ANSI cast error for example:
- `SELECT ... WHERE <filter>` returns the `'hr'` row, which the filter does
not accept, and
reports no error;
- `DELETE FROM ... WHERE <filter>` deletes the `'hr'` partition, which the
condition never
accepted.
The second is silent data loss.
The invariant was identified during the SPARK-58523 review, where
`fullyPushedFilterAttributes`
was described as the first thing in Spark to remove the post-scan
`FilterExec`. The two paths
above already removed it.
Keeping the pushed filter instead of propagating is not available here: by
the time `eval` fails,
the connector has accepted the predicate and Spark has already dropped the
filter, and Spark
never learns that an evaluation failed.
### Does this PR introduce _any_ user-facing change?
Yes. A query whose partition predicate cannot be evaluated now fails with
the underlying error
instead of silently returning rows the filter rejects, or deleting
partitions the condition never
accepted. The error is the one the same query produces without pushdown.
Queries that appear to
succeed on 4.2.0 or 4.3.0 may start failing; that failure is the correct
result and was
previously masked.
The same applies to a connector that passes a partition key whose width does
not match
`Table.partitioning()`. That now raises an internal error rather than being
silently treated as a
match.
Runtime filtering is unchanged.
### How was this patch tested?
New tests: a scan and a metadata-only DELETE whose partition predicate fails
to evaluate must
surface the error rather than keep the partition, and the DELETE must leave
both partitions in
place. Both fail without the fix. A unit test pins the split, asserting that
the same predicate
propagates by default and reports a match when constructed for a runtime
filter.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
