dongjoon-hyun commented on code in PR #60075:
URL: https://github.com/apache/spark/pull/60075#discussion_r4469896406
##########
docs/sql-migration-guide.md:
##########
@@ -74,6 +74,7 @@ license: |
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a user-defined scalar function, including a Data Source V2
function. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SQL_UDF_USAGE` on a SQL function. Previously, a write could
resolve the function name in its own session, so that it enforced another rule
or failed (SPARK-60777). To run these statements, change the condition to call
only built-in functions. Existing constraints that call a user-defined function
are not rejected.
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` when the
condition of a CHECK constraint calls an aggregate or window function,
including a user-defined aggregate function, `grouping`, and `grouping_id`.
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with its existing errors, such
as `INVALID_WHERE_CONDITION` and `WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE`.
Previously, these statements succeeded and every later write to the table
failed (SPARK-60800). To run these statements, remove the function from the
condition. Existing constraints that call these functions are not rejected.
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `GENERATOR_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a generator function, such as `explode`, `posexplode`, and
`inline`. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_GENERATOR.OUTSIDE_SELECT`. Previously, these statements succeeded
and every later write to the table failed (SPARK-60821). To run these
statements, remove the function from the condition. Existing constraints that
call a generator function are not rejected.
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` fail with
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` when the data type of a generation
expression names its struct fields differently from the column type, in name or
in order, for example `named_struct('y', a, 'x', b)` or `struct(a, b)` for a
`STRUCT<x: INT, y: INT>` column. Previously, such an expression was accepted,
although Spark lines up the struct fields by name when it computes the value of
a generated column of a Data Source V2 table, and by position when it checks a
provided value (SPARK-60857). To line up the fields by position, cast the
expression to the column type, for example `CAST(struct(a, b) AS STRUCT<x: INT,
y: INT>)`.
Review Comment:
Thank you, you're right. Both the auto-fill (SPARK-57644) and the capability
(SPARK-58311) are in 4.3.0, so a table that 4.3.0 created can already hold such
a row. I fixed the user-facing change section, and the migration guide entry
now says that such an existing table is not rejected, and that a write that
carries a row Spark auto-filled into it keeps failing the check.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala:
##########
@@ -120,6 +121,14 @@ case class GeneratedColumnExpression(
s"is incompatible with column data type
${targetDataType.simpleString}")
}
+ // Store assignment lines up the struct fields of an auto-filled value
with the column type by
+ // name, while the check of a provided value lines them up by position, so
the two must agree.
+ if (!DataType.equalsStructurallyByName(child.dataType, targetDataType,
resolver)) {
Review Comment:
Thanks. Agreed. Store assignment matches the fields under the writer's
resolver, so only an exact comparison lines up for every writer. I applied the
suggestion and added the case-differing expression to the test.
`named_struct('X', b, 'Y', a)` is now rejected, and I updated the table in the
description.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala:
##########
@@ -120,6 +121,14 @@ case class GeneratedColumnExpression(
s"is incompatible with column data type
${targetDataType.simpleString}")
}
+ // Store assignment lines up the struct fields of an auto-filled value
with the column type by
+ // name, while the check of a provided value lines them up by position, so
the two must agree.
+ if (!DataType.equalsStructurallyByName(child.dataType, targetDataType,
resolver)) {
Review Comment:
Thank you for checking it. You're right. `ALTER TABLE` and `CREATE TABLE ...
LIKE` do not run `validate`, so they skip all of its checks, not only this one,
and this predates the PR. I narrowed the description to `CREATE TABLE` and
`REPLACE TABLE`, and I'd like to handle `ALTER TABLE` on the fields of a
generated column in a separate JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
