txwyy123 opened a new issue, #59787: URL: https://github.com/apache/spark/issues/59787
### What changes were proposed in this pull request? Wrap wide output evaluation in join operators (`SortMergeJoinExec`, `BroadcastHashJoinExec`) and `HashAggregateExec` into separate methods to keep the generated `processNext()` method under HotSpot's 8,000-byte JIT compilation limit. ### Why are the changes needed? SPARK-60431 (PR #59678) splits wide `ProjectExec` outputs into per-column methods, fixing a 5.5x regression on 50-column projections. However, the same 8K-byte JIT cliff affects other operators that produce wide output: - **`SortMergeJoinExec`**: A join of two 50-column tables generates ~1,150 lines of Java in a single `processNext()` method (~17-25KB bytecode). HotSpot refuses to JIT-compile methods >8,000 bytes, so the hot inner loop runs interpreted. - **`BroadcastHashJoinExec`**: Same problem via the `HashJoin` trait's `codegenInner`/`codegenOuter`. - **`HashAggregateExec`**: Wide GROUP BY + many aggregate functions produces a similarly oversized `generateResultFunction`. Interestingly, `SortMergeJoinExec.codegenFullOuter` **already** wraps `consume()` in a separate method via `ctx.addNewFunction(consumeFullOuterJoinRow, ...)` — but the other join types (`codegenInner`, `codegenOuter`, `codegenSemi`, `codegenAnti`, `codegenExistence`) inline `consume()` directly into `processNext()`. This PR extends the proven `codegenFullOuter` pattern to all join types and to the other affected operators. ### Does this PR introduce _any_ user-facing change? No. This is a performance improvement only — query results are identical. Wide joins that previously ran 4-5x slower due to interpreter fallback now run at full JIT-compiled speed. ### How was this patch tested? - New benchmark: 50+50 column `SortMergeJoin` comparing before/after method sizes and wall-clock time - Existing `WholeStageCodegenSuite` tests pass - New test cases for wide inner join, wide outer join, wide aggregate output ### Related JIRAs - SPARK-21871: Original `hugeMethodLimit` awareness (Spark 2.3.0) - SPARK-33301: CASE WHEN branch splitting at 8K (Spark 4.4.0) - SPARK-60047: Split large non-CASE-WHEN expressions (proposed) - SPARK-60431: Split wide projection outputs (proposed, PR #59678) - **This ticket**: Split wide join/aggregate outputs (complementary to SPARK-60431) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
