pan3793 opened a new pull request, #58635:
URL: https://github.com/apache/spark/pull/58635
### What changes were proposed in this pull request?
Add a `RUNTIME_FILTER(relation)` join hint. It names a join side as the
runtime filter source:
Spark builds a runtime filter from that side and prunes the other side of
the join.
- One hint for both runtime filter kinds, and not a join strategy.
`HintInfo` gains a
`runtimeFilterSource` facet next to `strategy`, resolved with the same
relation matching as
join strategy hints, so it composes with `BROADCAST`, `MERGE`, etc. on the
same relation.
Spark picks the mechanism: partition pruning when the pruned side's join
key allows it, a
Bloom filter otherwise.
- The hint overrides cost estimates, not correctness. It waives the
selective-predicate search,
the size thresholds, the lineage and shuffle requirements, and DPP's
benefit estimate and
`reuseBroadcastOnly` gate. It keeps the join-type and simple-key
requirements, requires the
hinted side to be a repeatable source
(`JoinSelectionHelper.isRepeatableRuntimeFilterSource`),
and leaves the filter count and Bloom filter size limits in place. A
hinted side is never
itself pruned.
- The hint is never silently dropped. `InjectRuntimeFilter` warns with the
reason whenever it is
not applied, including when Bloom filters are disabled. Hinting both sides
is ambiguous: Spark
warns and falls back to the heuristics.
This supersedes SPARK-32842 (#29709, a DPP-only hint) and SPARK-42064
(#39571,
`BLOOM_FILTER_JOIN` as a `JoinStrategyHint`).
### Why are the changes needed?
Spark's runtime filter heuristics depend on statistics and a small set of
recognized plan shapes.
When statistics are missing or misleading, or the filter source is an
aggregate, a subquery, or
a join, no runtime filter is built even though the user knows the pruning
would pay off. Earlier
attempts to relax the DPP cost model automatically (SPARK-34884) were
reverted because the
estimates are unreliable, and `reuseBroadcastOnly` applies to every join in
the session. A hint
makes the choice explicit for one join, the same way join strategy hints
override join selection.
### Does this PR introduce _any_ user-facing change?
Yes. A new SQL hint `RUNTIME_FILTER` and the corresponding
`Dataset.hint("runtime_filter", ...)`
name are recognized. Previously the name was reported as an unrecognized
hint and ignored. The
hint parameter error message now reads "Join hint parameter ..." instead of
"Join strategy hint
parameter ...".
### How was this patch tested?
New tests in `InjectRuntimeFilterSuite` (which heuristics the hint waives,
which requirements it
keeps, DPP deduplication and the non-pushable DPP case, composition with
strategy hints,
ambiguous and unapplicable hints with their warning text, subquery and
parameter-less forms),
`DynamicPartitionPruningSuite` (standalone DPP subquery under the hint,
broadcast reuse,
partition-key requirement, both-sides hint), and `ResolveHintsSuite`
(resolution and merging).
Existing `JoinHintSuite` and hint-related suites pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
