dongjoon-hyun opened a new pull request, #60035:
URL: https://github.com/apache/spark/pull/60035
### What changes were proposed in this pull request?
This PR fixes a correctness issue: an auto-filled generated column of a Data
Source V2 table read a column referenced in a lambda function, such as `i` in
`transform(a, x -> x + i)`, from the query before the store assignment cast, so
it could silently store a wrong value.
#60008 fixed the same issue for a field of another column, such as `s.f`.
This PR fixes it for a column referenced in a lambda function, which was found
in the review of #60008.
`TableOutputResolver.resolveGenerationExprReferences` replaces the column
references in the generation expression with the stored values. The parser
turns every name in a lambda function into `UnresolvedNamedLambdaVariable`, so
the rewrite did not see a column referenced in a lambda function, and
`ResolveLambdaVariables` later turned it back into `UnresolvedAttribute`, which
resolved against the query.
Now the rewrite keeps the names declared by the enclosing lambda functions.
An `UnresolvedNamedLambdaVariable` that none of them declares refers to a
column, so it is replaced with the stored value in the same way as an
`UnresolvedAttribute`, including a field such as `s.f`. The rewrite walks only
the body of a lambda function, not its arguments, which are the declared
variables.
### Why are the changes needed?
A write computes the generated value from a value that is not the one it
stores. The row silently stores a generated value that does not match its own
column, and a later write of that row fails. These are the three examples from
the review of #60008:
```sql
CREATE TABLE t (arr ARRAY<INT>, i INT,
g ARRAY<INT> GENERATED ALWAYS AS (transform(arr, x -> x * i))) USING foo;
INSERT INTO t (arr, i) SELECT array(10), 5.7D;
-- Issue 1: g should be [50], but it stores [57]
INSERT INTO t SELECT * FROM t;
-- Issue 2: this should succeed, but it fails with CHECK_CONSTRAINT_VIOLATION
```
- `transform(arr, x -> x * s.f)` on `s STRUCT<f: INT>` stores `[57]` instead
of `[50]` for `INSERT INTO t (arr, s) SELECT array(10), named_struct('f',
5.7D)`.
- `aggregate(arr, 0, (acc, x) -> acc + x * i)` on `i INT` fails with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` for `INSERT INTO t (arr, i) SELECT
array(10), 5.7D`, because `i` is the `DOUBLE` of the query. It should store
`50`.
The same gap affects these writes:
- A stored value that differs without a cast. `transform(a, x -> concat(x,
c, '|'))` on `c CHAR(3)` stored `pa|` beside the stored `a `.
- `map_filter(m, (k, v) -> v > i * 10)` stored an empty map for `map('k',
52)` and `i` of `5.7D`, instead of `{k -> 52}`.
- A column filled with its default value. It failed with `UNRESOLVED_COLUMN`
when `spark.sql.lateralColumnAlias.enableImplicitResolution` is `false`.
### Does this PR introduce _any_ user-facing change?
Yes. This fixes a correctness bug in a feature released in Spark 4.3.0. An
auto-filled generated column now reads a column referenced in a lambda function
from the stored value, so the three examples above store `[50]`, `[50]`, and
`50`, and the writes that failed above now succeed. Rows that are already
stored keep their values. The SQL migration guide describes this.
### How was this patch tested?
Added a test to `GeneratedColumnWriteSuite`. Without this fix, it fails with
`[[1],5,[58],[[1]]]` instead of `[[1],5,[51],[[1]]]`. The test also checks that
a lambda variable with the name of a column, `i` in `transform(a, i ->
transform(a, x -> i))`, still reads the lambda variable.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
