txwyy123 opened a new issue, #59911:
URL: https://github.com/apache/spark/issues/59911
### Are you reporting a bug, or is this a request for an enhancement?
Bug report.
### What are you trying to do?
Enable whole-stage codegen for queries with more than 100 output columns by
raising `spark.sql.codegen.maxFields` above the default of 100.
### Describe the bug
When `spark.sql.codegen.maxFields` is raised above 100 (e.g. to 200),
SortMergeJoin codegen produces code that **fails to compile** with:
```
org.codehaus.janino.CompileException: Unknown variable or type
"smj_bufferedRow_1"
```
The root cause is Spark's class-size splitting mechanism
(`addNewFunctionToClass`). When the generated code for a single class exceeds
~64KB of bytecode, Spark moves methods into **nested inner classes**. However,
the moved methods still reference **local variables** (`smj_bufferedRow_1`,
`smj_value_4`, etc.) declared in the outer method's scope. Janino (the runtime
Java compiler) correctly rejects these cross-scope references.
This bug is **latent in stock Spark** because `maxFields` defaults to 100,
which disables codegen entirely for wide schemas before the class-size split
triggers. The bug only surfaces when a user raises `maxFields` hoping to get
codegen benefits on wider queries.
### Reproduction
```scala
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
spark.conf.set(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key, "-1") // force SMJ
spark.conf.set(SQLConf.WHOLESTAGE_MAX_NUM_FIELDS.key, "200") // enable wide
codegen
val numCols = 60 // 60+60 = 120 output columns
val left = (0 until numCols).foldLeft(spark.range(100).toDF("id")) { (df, i)
=>
df.withColumn(s"l_col$i", col("id") + i)
}
val right = (0 until numCols).foldLeft(spark.range(100).toDF("id")) { (df,
i) =>
df.withColumn(s"r_col$i", col("id") + i)
}
// This throws CompileException: Unknown variable or type "smj_bufferedRow_1"
left.join(right, left("id") === right("id")).collect()
```
The exact column threshold depends on data types and whether a join
condition is present, but typically triggers around 50-70 columns per side
(100-140 total output columns).
### Expected behavior
The query should either:
1. Compile and execute correctly with codegen enabled, or
2. Gracefully fall back to the Volcano iterator model (as it does for other
codegen compilation failures)
### Environment
- Apache Spark master (4.0.0-SNAPSHOT)
- Also reproducible on Spark 3.5.x
### Analysis
The issue is distinct from the 8K JIT compilation limit (SPARK-59787). This
is about the **64KB class-size limit** causing code to move to inner classes
that can't see outer-scope locals.
Two possible fixes:
1. **Promote local variables to class fields** before the class-size split
can move their consumers — this is what #59841 does for the column extraction
code specifically
2. **Detect the cross-scope reference** in the code generator and fall back
gracefully to non-codegen execution
Related: #59787, #59841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
