Szehon Ho created SPARK-61495:
---------------------------------
Summary: Use modeled DSv2 expressions for generated column writes
Key: SPARK-61495
URL: https://issues.apache.org/jira/browse/SPARK-61495
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Szehon Ho
Use modeled DSv2 expressions for generated column writes
GenerationExpression accepts either SQL text or a portable DSv2 expression.
Spark can validate an explicitly supplied generated value using a modeled
expression, but its auto-fill path currently reads only SQL text. The
validation path also prefers SQL when both forms are present, while default
values and CHECK constraints prefer the modeled form.
For a connector table declaring a INT and b INT with a modeled a + 1 expression
and GENERATE_COLUMN_VALUES_ON_WRITE, inserting only a = 1 should store (1, 2).
Today the missing b is treated as an ordinary missing column and the write
fails its generated-column check.
Extend write resolution to convert generation expressions directly to Catalyst
and compute them from stored input values after assignment casts. Share the
conversion between auto-fill and validation. Prefer the modeled form, falling
back to SQL only when the model is absent or unsupported by the converter.
Report analysis and evaluation failures after conversion without changing
representations.
This changes results when both representations exist and differ, including
implicit casts preserved in the modeled expression by CREATE TABLE. For
example, coalesce(s, 1) defined under a non-ANSI session generates '007' for s
= '007' from its modeled string fallback, where reparsing SQL previously
produced '7'.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]