dongjoon-hyun opened a new pull request, #60081:
URL: https://github.com/apache/spark/pull/60081
### What changes were proposed in this pull request?
This PR fixes a correctness issue: a write to a DSv2 table stores a wrong
value in an auto-filled generated column, and checks a provided value against a
wrong value, when a function in the SQL path of the write shadows a built-in
function of the generation expression. It also makes the DDL reject a
generation expression that calls a V2 function with a magic method, whose name
a write also resolves in its own session.
Both reuse what SPARK-60777 added for CHECK constraints. This PR moves that
code to a new object, `StoredExpressionFunctions`, so that CHECK constraints
and generated columns share it.
1. `StoredExpressionFunctions.qualifyBuiltinFunctions` rewrites an
unqualified `UnresolvedFunction` to `system.builtin.<name>` when `<name>` is a
built-in function. `StoredExpressionText` now applies it to every text that it
parses. Every read of the stored text of a generation expression goes through
it: when a write auto-fills a missing generated column, when it checks a
provided value, and when it reads the text under its own session.
2. `CheckAnalysis` rejects a generation expression with
`StoredExpressionFunctions.findUserDefinedFunctionCall`, which
`CheckConstraint.failIfCallsUserDefinedFunction` now calls too. The previous
check matched only `UserDefinedExpression`, which misses `Invoke` and
`StaticInvoke`: V2 functions implementing an `invoke` method or a static
`invoke` method. This PR also corrects the comment above the check, which said
that all V2 functions are `UserDefinedExpression`.
3. The SQL migration guide describes these changes.
The check is in the DDL only, so an existing generated column that calls a
V2 function keeps working. Like the rewrite for CHECK constraints, this one
does not cover a built-in function called through an `IDENTIFIER` clause, such
as `IDENTIFIER('upper')(s)`, or a function injected with
`SparkSessionExtensions.injectFunction`.
### Why are the changes needed?
This is a correctness issue. A generation expression is stored as SQL text,
and a write parses that text again. So function names are resolved in the
session of the write, not in the session of the DDL. A function in the SQL path
can shadow a built-in function, and the write silently stores a wrong value:
```sql
-- testcat.ns.upper is a V2 function that always returns 'OK', and
spark.sql.path.enabled is true.
CREATE TABLE testcat.t (s STRING, g STRING GENERATED ALWAYS AS (upper(s)))
USING foo;
SET PATH = testcat.ns, system.builtin;
-- Issue 1: It should store g = 'X', but it stores g = 'OK'.
INSERT INTO testcat.t (s) VALUES ('x');
-- Issue 2: It should succeed, but it fails with CHECK_CONSTRAINT_VIOLATION.
INSERT INTO testcat.t VALUES ('y', 'Y');
-- Issue 3: It should fail with CHECK_CONSTRAINT_VIOLATION, but it stores g
= 'OK'.
INSERT INTO testcat.t VALUES ('z', 'OK');
```
In Apache Spark
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27), a
temporary function does the same with
`spark.sql.functionResolution.sessionOrder=first`. SPARK-60757 reads the text
under fixed settings, which stops the temporary function, but not the SQL path.
Both still shadow a built-in function when a write reads the text under its own
session: for a text that the fixed settings cannot resolve, and with
`spark.sql.legacy.useCurrentConfigsForGeneratedColumn=true`.
A generation expression can call only built-in functions, and `CREATE TABLE`
already rejected a V2 function implementing `produceResult`. The same function
implementing `invoke` passed the DDL, and then a write resolved its name in its
own session.
### Does this PR introduce _any_ user-facing change?
Yes.
- A write always binds a built-in function of a parsed generation expression
to `system.builtin`, regardless of its SQL path and
`spark.sql.functionResolution.sessionOrder`. Before, the value that it filled
in or checked could come from another function with the same name. Apache Spark
4.3.0, which added the auto-fill, already stores such values. Rows that are
already stored keep their values.
- Before, `CREATE TABLE` and `REPLACE TABLE` accepted a generation
expression that calls a V2 scalar function with a magic `invoke` method. After,
they fail as they do for other user-defined functions:
```
[UNSUPPORTED_EXPRESSION_GENERATED_COLUMN] Cannot create generated column g
with generation expression testcat.ns.strlen(s) because failed to resolve
`strlen_magic` to a built-in function. SQLSTATE: 42621
```
Existing generated columns that call a V2 function are not rejected, and
their functions are still resolved in the session of the write.
### How was this patch tested?
Pass the CIs with the newly added test cases in `GeneratedColumnWriteSuite`:
- `CREATE TABLE` rejects a V2 function implementing an `invoke` method or a
static `invoke` method.
- `SET PATH` does not shadow a built-in function, both when a write
auto-fills a value and when it checks a provided value.
Both test cases fail without the fix.
```
build/sbt 'sql/testOnly *GeneratedColumnWriteSuite *CheckConstraintSuite
*DataSourceV2DataFrameSuite'
build/sbt 'sql/testOnly *DataSourceV2SQLSuiteV1Filter -- -z "generation
expression"'
build/sbt 'catalyst/testOnly *GeneratedColumnExpressionSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
