pratham76 opened a new pull request, #59700:
URL: https://github.com/apache/spark/pull/59700
### What changes were proposed in this pull request?
We add a `SKEW_JOIN` join hint and a new physical operator `SkewKeyJoinExec`
for joins where a key is frequent on both sides.
Syntax: `/*+ SKEW_JOIN(t(col1, col2)) */` (`SKEWJOIN` is an alias). The
columns must be equi-join keys of the hinted relation.
* **Hint:** `SKEW_JOIN` is a new `JoinStrategyHint`, and `HintInfo` gets a
`skewColumns` field. `ResolveJoinStrategyHints` reads `t(col)` (parsed as a
function call, so no grammar change) and attaches the columns to the relation
`t`.
* **Planning:** `JoinSelection` plans an equi-join with the hint as a
`SkewKeyJoinExec`. If the join cannot be split, the hint is ignored with the
usual hint warning and the join is planned as without the hint. Supported are
inner joins, left outer joins hinted on the left side, and right outer joins
hinted on the right side, with join keys of integral, decimal, boolean, date,
timestamp or binary-collated string types that are the same on both sides.
* **Execution:** `SkewKeyJoinExec` samples the hinted side to find the hot
keys and splits the join in two:
* the rows of the hot keys of both sides are joined with a
`BroadcastHashJoinExec`, streaming the hinted side in its existing
partitioning, and
* all the other rows are joined with a `SortMergeJoinExec`.
The results are unioned. Null keys are never hot and are handled by the
sort merge join. If no key is hot, only the sort merge join runs. Since each
side is read twice, a join with non-deterministic expressions is rejected with
the new error `INVALID_SKEW_JOIN_HINT` (as are a column that is not in the
relation and a malformed hint).
* **Hot keys:** a key is hot if its estimated number of rows (from the
sample) is at least `spark.sql.skewJoinHint.minHotKeyRows` and at least the
average rows of a shuffle partition (estimated rows of the hinted side divided
by `spark.sql.shuffle.partitions`). At most `spark.sql.skewJoinHint.maxHotKeys`
keys are used, the most frequent first.
* **Configs:** `spark.sql.skewJoinHint.sampleFraction` (0.1),
`spark.sql.skewJoinHint.minHotKeyRows` (1000),
`spark.sql.skewJoinHint.maxHotKeys` (1000).
### Why are the changes needed?
In a fact-to-fact join, a key that is frequent on both sides sends the
Cartesian product of its rows to a single task, e.g. 5,000 x 10,000 = 50M
output rows for one key. When that key is spread thinly over many mappers, all
the shuffle blocks are small, so `OptimizeSkewedJoin`, which only looks at the
sizes of the shuffle blocks, does not detect it. Splitting the partition and
replicating the other side would not help either, as the problem is the output
of a key and not the size of a block. Splitting the hot keys out of the join
and broadcasting them spreads their output over all the tasks that hold their
rows.
### Does this PR introduce _any_ user-facing change?
Yes. A new `SKEW_JOIN` join hint, the new error condition
`INVALID_SKEW_JOIN_HINT`, and three new configs. Without the hint, nothing
changes.
### How was this patch tested?
Added `SkewKeyJoinSuite`, which checks that the results are the same as
without the hint, with AQE on and off, for inner joins hinted on either side,
multiple keys, a subset of the join keys, non equi-join conditions, left and
right outer joins, and null keys. It also covers no hot keys, the cap on the
number of hot keys, sampling, the cases where the hint is ignored with a
warning (unsupported join type, hinted side that cannot be split, column that
is not a join key, different key types, double keys), and the error cases.
Also tested with existing UT
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
