dongjoon-hyun opened a new pull request, #60008:
URL: https://github.com/apache/spark/pull/60008
### What changes were proposed in this pull request?
This PR fixes `TableOutputResolver.resolveGenerationExprReferences` so that
a multi-part reference in the generation expression of an auto-filled generated
column of a Data Source V2 table, such as `s.f`, reads the value stored in the
column `s`.
The first name part is replaced with the stored column, and the remaining
parts become `UnresolvedExtractValue`, which is what the parser produces for a
field of an expression. Previously, only single-part references were replaced,
so `s.f` resolved against the column of the query, before the store assignment
cast.
### Why are the changes needed?
A write computes the generated value from a value that is not the one it
stores. The row silently stores a generated value that does not match its own
column, and a later write of that row fails.
```sql
CREATE TABLE t (id INT, s STRUCT<f: INT>, i INT,
b INT GENERATED ALWAYS AS (s.f * 10), d INT GENERATED ALWAYS AS (i * 10))
USING foo;
INSERT INTO t (id, s, i) SELECT 1, named_struct('f', 5.7D), 5.7D;
-- stored: [1,[5],5,57,50], where b should be 50
INSERT INTO t SELECT * FROM t; -- CHECK_CONSTRAINT_VIOLATION
```
`d` above and `s['f']` already read the stored value.
The same gap affects these writes:
- A field of a nested struct (`s.n.f`), a map key (`m.k`), and a field of an
array of structs (`a.f`).
- A stored value that differs without a cast. `concat(s.f, '|')` on a
`CHAR(3)` field stored `a|` beside the stored `a `.
- A field the query does not provide. `INSERT WITH SCHEMA EVOLUTION` filling
the missing field failed with `FIELD_NOT_FOUND` when
`spark.sql.insertNestedTypeCoercion.enabled` is `true`.
- A column filled with its default value. It failed with `UNRESOLVED_COLUMN`
when `spark.sql.lateralColumnAlias.enableImplicitResolution` is `false`.
### Does this PR introduce _any_ user-facing change?
Yes. This fixes a correctness bug in a feature released in Spark 4.3.0. An
auto-filled generated column now reads a field of another column from the
stored value, so the query above stores `50` instead of `57`, and the writes
that failed above now succeed. Rows that are already stored keep their values.
The SQL migration guide describes this.
### How was this patch tested?
Added a test to `GeneratedColumnWriteSuite`. Without this fix, it fails with
`[[5],57]` instead of `[[5],50]`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
