dongjoon-hyun opened a new pull request, #59969:
URL: https://github.com/apache/spark/pull/59969
### What changes were proposed in this pull request?
This PR aims to pass `functionName`, `inputTypesString`, and `outputType` of
`ScalaUDF` to the generated code via `ctx.addReferenceObj` instead of embedding
them into Java string literals. The interpreted path is unchanged.
### Why are the changes needed?
These values hold text from the query, such as the UDF name and the field
names of struct types. The generated code puts them into Java string literals
without escaping, so a double quote or an invalid escape sequence makes the
generated class fail to compile.
```scala
spark.udf.register("rowSize", (r: Row) => r.size)
sql("""SELECT rowSize(named_struct('fieldName"b', id)) AS r FROM
range(2)""").collect()
```
This has been the case since SPARK-33541 (Apache Spark 3.2.0). Before that,
the error message was passed via `ctx.addReferenceObj("errMsg", ...)`.
References are preferred to escaping because the text stays out of the
generated source:
- A very long name does not hit the constant string limit of `javac`.
- Queries that differ only in these names can share a cached generated class.
### Does this PR introduce _any_ user-facing change?
Yes, for UDFs whose name or input and output types contain these characters.
- Previously, the generated code failed to compile. Queries still returned
correct results through the fallback, but compile errors were logged and
whole-stage codegen was disabled for the plan. After this PR, the generated
code compiles.
- Previously, a valid Java escape sequence such as `\t` in a name compiled,
but the `FAILED_EXECUTE_UDF` error showed the unescaped text, unlike the
interpreted path. After this PR, both paths report the same text.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `ScalaUDFSuite`: A UDF whose name, input type, and output type contain `"`
and `\q`. The generated and interpreted paths must report the same
`FAILED_EXECUTE_UDF` parameters, and none of the three values may appear in the
generated code. The test fails if any one of the three values is put back into
a string literal.
- `UDFSuite`: The SQL query above must return the expected rows, and the
field name must not appear in the whole-stage generated code. The test fails
without the source change.
The following suites also pass locally.
```bash
build/sbt 'catalyst/testOnly *ScalaUDFSuite *CodeGenerationSuite
*CodeCompilerSuite'
```
```bash
build/sbt 'sql/testOnly org.apache.spark.sql.UDFSuite
*WholeStageCodegenSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
