hhr293 opened a new pull request, #12513:
URL: https://github.com/apache/gluten/pull/12513
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
## Description
Enable BuildLeft for LeftSemi `ShuffledHashJoin` on the Velox backend,
guarded by a
QueryStagePrepRule and gated behind an opt-in switch.
### Motivation
`OffloadJoin.getShjBuildSide` -> `getOptimalBuildSide` currently cannot
choose BuildLeft
for `LeftSemi` on Velox because `supportHashBuildJoinTypeOnLeft` returns
`false` for
`LeftSemi`. This forces the (typically larger) fact table onto the build
side, producing
oversized hash tables. Velox already supports `kRightSemiFilter` via
Substrait
`JOIN_TYPE_RIGHT_SEMI`; the referenced upstream restriction (Velox issue
#9980) has been
resolved. Removing the restriction lets the optimizer place the smaller
side on build.
BuildLeft, however, is only profitable when:
- the shuffle is large enough to absorb streamed-probe overhead on the
right side, and
- right/left is skewed enough that hash-build savings on the left dominate
that overhead,
- and the right side is not partition-skewed (AQE `OptimizeSkewedJoin`
cannot split the
probe side of a `ShuffledHashJoin`).
To ship the optimization safely, this PR adds a `QueryStagePrepRule` that
inspects
shuffle stats at AQE re-plan time and forces BuildRight when any of the
three
conditions above fail. The whole feature is behind an opt-in switch and
disabled by
default.
## Commits
1. **[VL] Enable BuildLeft for LeftSemi ShuffledHashJoin**
Adds `LeftSemi => true` to `supportHashBuildJoinTypeOnLeft` and fixes
the
backend-specific post-join projection index in `JoinUtils` via a new
`BackendSettingsApi.leftSemiBuildLeftOutputsBuildOnly` capability flag
(Velox = true, ClickHouse = false).
2. **[VL] Guard LeftSemi BuildLeft against probe-side partition skew**
New shared utility `ShuffleSkewDetector` + new Velox-only
QueryStagePrepRule
`LeftSemiBuildLeftGuardRule` that tags LeftSemi joins with
`RewriteJoin.ForceShjBuildRightTag` when the right-side shuffle is
skewed
(per Spark's own AQE `skewedPartitionFactor` /
`skewedPartitionThresholdInBytes`).
`RewriteJoin` and `OffloadJoin` honor the tag to fall back to
BuildRight.
3. **[VL] Guard LeftSemi BuildLeft with an opt-in switch and size floor**
Adds three Gluten configs and reorders the guard checks in ascending
cost:
-
`spark.gluten.sql.columnar.shuffledHashJoin.leftSemi.buildLeft.enabled` (bool,
default **false**)
-
`spark.gluten.sql.columnar.shuffledHashJoin.leftSemi.buildLeft.minRightBytes`
(bytes, default **10GB**, mirrors Spark's `applicationSideScanSizeThreshold`)
-
`spark.gluten.sql.columnar.shuffledHashJoin.leftSemi.buildLeft.minRightToLeftRatio`
(double, default **10.0**)
## Behavior when disabled
With `leftSemi.buildLeft.enabled=false` (the default),
`VeloxBackend.supportHashBuildJoinTypeOnLeft`
rejects `LeftSemi` outright and the guard rule short-circuits to `plan`
without walking it,
so behavior is byte-identical to a Gluten build without this patch.
## Configuration reference
| conf | default | rationale |
|------|---------|-----------|
| `leftSemi.buildLeft.enabled` | `false` | Opt-in. This optimization is
deployment-sensitive; shipping enabled-by-default would risk regressions on
workloads that do not fit the profitable region. |
| `leftSemi.buildLeft.minRightBytes` | `10GB` | Aligned with Spark's own
`spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold`
(10GB) -- the same "big enough that a costly optimization is worth doing" gate.
|
| `leftSemi.buildLeft.minRightToLeftRatio` | `10.0` | Conceptually
analogous to Spark's `spark.sql.shuffledHashJoinFactor`, applied at build-side
selection within SHJ. 10x guarantees a one order-of-magnitude size gap between
the two sides so BuildLeft's hash-build savings dominate the streamed-probe
overhead on the right. |
## Benchmark results
TPC-DS and TPC-H on Velox backend, 36 executors * 8 cores, off-heap 32G,
HDFS shuffle.
Patch mode: `enabled=true`, `minRightBytes=10GB` (default),
`minRightToLeftRatio=10.0` (default).
Each cell is the mean of 3 full sweeps.
### Total latency
| Workload | Patch OFF (mean) | Patch ON (mean) | Delta |
|----------|-----------------:|----------------:|------:|
| TPC-DS 1TB | 534.17 s | 542.42 s | **+1.55%** (within noise) |
| **TPC-DS 6TB** | 1417.33 s | 1368.92 s | **-3.42%** ✅ |
| TPC-H 1TB | 245.72 s | 244.43 s | -0.53% (within noise) |
| TPC-H 6TB | 1099.28 s | 1088.49 s | -0.98% |
### TPC-DS 6TB impact
| Query | Off | On | Delta |
|-------|----:|---:|------:|
| **q95** | 95.12 s | 57.48 s | **-39.6%** ✅ |
| **q94** | 9.16 s | 6.49 s | **-29.1%** ✅ |
| **q16** | 17.27 s | 12.64 s | **-26.8%** ✅ |
### TPC-H 6TB impact
| Query | Off | On | Delta |
|-------|----:|---:|------:|
| **q4** | 30.70 s | 21.90 s | **-28.7%** ✅ |
| q20 | 21.01 s | 19.41 s | -7.6% |
- TPC-DS 1TB with patch OFF and patch ON -- no regression (`total` within
2% of baseline)
- TPC-DS 6TB with patch ON -- expected -3.4% total latency improvement
- TPC-H 6TB with patch ON -- expected ~-1% total improvement, no per-query
regression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
