peter-toth commented on code in PR #60075:
URL: https://github.com/apache/spark/pull/60075#discussion_r4470767913
##########
docs/sql-migration-guide.md:
##########
@@ -74,6 +74,7 @@ license: |
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a user-defined scalar function, including a Data Source V2
function. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SQL_UDF_USAGE` on a SQL function. Previously, a write could
resolve the function name in its own session, so that it enforced another rule
or failed (SPARK-60777). To run these statements, change the condition to call
only built-in functions. Existing constraints that call a user-defined function
are not rejected.
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` when the
condition of a CHECK constraint calls an aggregate or window function,
including a user-defined aggregate function, `grouping`, and `grouping_id`.
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with its existing errors, such
as `INVALID_WHERE_CONDITION` and `WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE`.
Previously, these statements succeeded and every later write to the table
failed (SPARK-60800). To run these statements, remove the function from the
condition. Existing constraints that call these functions are not rejected.
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `GENERATOR_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a generator function, such as `explode`, `posexplode`, and
`inline`. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_GENERATOR.OUTSIDE_SELECT`. Previously, these statements succeeded
and every later write to the table failed (SPARK-60821). To run these
statements, remove the function from the condition. Existing constraints that
call a generator function are not rejected.
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` fail with
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` when the data type of a generation
expression names its struct fields differently from the column type, in name,
including letter case, or in order, for example `named_struct('y', a, 'x', b)`,
`named_struct('X', b, 'Y', a)`, or `struct(a, b)` for a `STRUCT<x: INT, y:
INT>` column. Previously, such an expression was accepted, although Spark lines
up the struct fields by name, with the case sensitivity of the writing session,
when it computes the value of a generated column of a Data Source V2 table, and
by position when it checks a provided value (SPARK-60857). To line up the
fields by position, cast the expression to the column type, for example
`CAST(struct(a, b) AS STRUCT<x: INT, y: INT>)`. Existing tables with such an
expression are not rejected, and a write that carries a row that Spark
auto-filled into one of them keeps failing the check.
Review Comment:
**Finding 3.** The last sentence is the one I suggested in finding 1, and it
holds only for fields of the same type in another order. With
`GeneratedColumnExpression.scala` from this PR's base, so that `CREATE TABLE`
accepts the expression as 4.3.0 does, I auto-filled a row with `INSERT INTO t
(a, b) VALUES (1, 2)` and wrote it again with `INSERT INTO t SELECT * FROM t`:
| Generation expression | Column type | Rewrite |
|---|---|---|
| `named_struct('y', a, 'x', b)` | `STRUCT<x: INT, y: INT>` |
`CHECK_CONSTRAINT_VIOLATION` |
| `named_struct('y', a, 'x', b)` | `STRUCT<x: BIGINT, y: BIGINT>` |
`DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES` |
| `map(a, named_struct('y', a, 'x', b))` | `MAP<INT, STRUCT<x: INT, y:
INT>>` | `DATATYPE_MISMATCH.INVALID_ORDERING_TYPE` |
| `named_struct('X', b, 'Y', a)` | `STRUCT<x: INT, y: INT>` | passes, in a
case-insensitive session |
So with the second and third, every write that provides the column fails
during analysis, not only one that carries an auto-filled row. The fourth keeps
working. For `struct(a, b)`, auto-fill already fails with `CANNOT_FIND_DATA`,
as the description's table shows, so there is no auto-filled row to carry. A
sentence that names no single outcome holds for all of them:
```suggestion
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` fail with
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` when the data type of a generation
expression names its struct fields differently from the column type, in name,
including letter case, or in order, for example `named_struct('y', a, 'x', b)`,
`named_struct('X', b, 'Y', a)`, or `struct(a, b)` for a `STRUCT<x: INT, y:
INT>` column. Previously, such an expression was accepted, although Spark lines
up the struct fields by name, with the case sensitivity of the writing session,
when it computes the value of a generated column of a Data Source V2 table, and
by position when it checks a provided value (SPARK-60857). To line up the
fields by position, cast the expression to the column type, for example
`CAST(struct(a, b) AS STRUCT<x: INT, y: INT>)`. Existing tables with such an
expression are not rejected, and writes to them behave as before.
```
The user-facing section of the description has the same sentence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
