dongjoon-hyun opened a new pull request, #59965: URL: https://github.com/apache/spark/pull/59965
### What changes were proposed in this pull request? This PR makes a write read the stored text of a generation expression under fixed settings, instead of the settings of the session that writes. - `StoredExpressionText` is a new helper that defines those settings: the default `SQLConf`, the configs that `Analyzer.retainResolutionConfigsForAnalysis` keeps, and ANSI enabled. The session time zone and case sensitivity stay with the session, because the settings of the DDL are not stored and pinning them would silently change existing tables. - `TableOutputResolver`, which auto-fills a generated column, and `ResolveTableConstraints.buildGenerationCatalystExpression`, which validates a provided value when the V2 expression cannot be converted back, parse the text under those settings. - `ResolveOutputRelation` also analyzes the auto-filled columns under those settings, the same way `ResolveSQLFunctions` analyzes a SQL function body. Implicit casts and the ANSI behavior of functions are decided during analysis, so parsing alone would leave them to the writer. - `GeneratedColumnExpression.validate` rejects a generation expression that reads differently under those settings than in the session that creates the table, with a new reason of `UNSUPPORTED_EXPRESSION_GENERATED_COLUMN`. The comparison uses the ANSI value of the session, so it is only about how the text reads. ### Why are the changes needed? Generated column values on V2 table writes were added by SPARK-57644 and first released in Apache Spark [v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27). The value of a generated column depends on the session that writes it, and a wrong value is stored silently. ```sql CREATE TABLE t (a STRING, b STRING GENERATED ALWAYS AS (concat(a, 'x\\y'))); INSERT INTO t (a) VALUES ('default'); -- b = defaultx\y SET spark.sql.parser.escapedStringLiterals=true; INSERT INTO t (a) VALUES ('escaped'); -- b = escapedx\\y INSERT INTO t (a, b) SELECT 'p', concat('p', 'x', chr(92), 'y'); -- CHECK_CONSTRAINT_VIOLATION INSERT INTO t (a, b) SELECT 'q', concat('q', 'x', chr(92), chr(92), 'y'); -- succeeds ``` ```sql CREATE TABLE t (a STRING, b BIGINT GENERATED ALWAYS AS (a + 1)); SET spark.sql.ansi.enabled=false; INSERT INTO t (a) VALUES ('abc'); -- b = NULL INSERT INTO t (a) VALUES ('9007199254740993'); -- b = 9007199254740992 ``` After this PR, the first example stores `escapedx\y`, accepts `p`, and rejects `q`. The second example fails with `CAST_INVALID_INPUT` for `abc` and stores `9007199254740994`. ### Does this PR introduce _any_ user-facing change? Yes, compared to Apache Spark 4.3.0. - The settings of the writer no longer change an auto-filled value through the parser settings, the ANSI mode, or a temporary function that replaces a built-in function. - A write with `spark.sql.ansi.enabled=false` now fails with the ANSI error of a generation expression instead of storing `NULL`, also for a table created with that setting. For example, `b DOUBLE GENERATED ALWAYS AS (a + 1)` with `a = '1.5'` used to store `2.5` when both the DDL and the write ran with that setting, and now fails with `CAST_INVALID_INPUT`. - `CREATE TABLE` and `REPLACE TABLE` reject a generation expression that the session reads differently. For example, with `spark.sql.parser.escapedStringLiterals=true`, `concat(a, 'x\\y')` is rejected, while `concat(a, 'xy')` is accepted. With `spark.sql.ansi.enforceReservedKeywords=true`, a bare `current_date` is rejected, and `curdate()` is accepted. The following still depend on the session that writes: - The analysis of the validation of a provided value, when the V2 expression cannot be converted back. - The functions found through `SET PATH`. - The session time zone, for example for `hour(ts)`. - The settings that an expression reads during evaluation, such as `spark.sql.legacy.timeParserPolicy` and `spark.sql.legacy.castComplexTypesToString.enabled`. ### How was this patch tested? Added test cases to `GeneratedColumnWriteSuite`, updated one test case in `DataSourceV2DataFrameSuite`, and changed the placeholder text in `GeneratedColumnExpressionSuite`. Eleven of the new test cases fail without this PR. Ran locally: ```bash build/sbt 'catalyst/testOnly *GeneratedColumnExpressionSuite' build/sbt 'sql/testOnly *GeneratedColumnWriteSuite *DataSourceV2DataFrameSuite' SPARK_ANSI_SQL_MODE=false build/sbt 'sql/testOnly *GeneratedColumnWriteSuite *DataSourceV2DataFrameSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
