[
https://issues.apache.org/jira/browse/SPARK-61495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61495:
-----------------------------------
Labels: pull-request-available (was: )
> Use modeled DSv2 expressions for generated column writes
> --------------------------------------------------------
>
> Key: SPARK-61495
> URL: https://issues.apache.org/jira/browse/SPARK-61495
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> Use modeled DSv2 expressions for generated column writes
> GenerationExpression accepts either SQL text or a portable DSv2 expression.
> Spark can validate an explicitly supplied generated value using a modeled
> expression, but its auto-fill path currently reads only SQL text. The
> validation path also prefers SQL when both forms are present, while default
> values and CHECK constraints prefer the modeled form.
> For a connector table declaring a INT and b INT with a modeled a + 1
> expression and GENERATE_COLUMN_VALUES_ON_WRITE, inserting only a = 1 should
> store (1, 2). Today the missing b is treated as an ordinary missing column
> and the write fails its generated-column check.
> Extend write resolution to convert generation expressions directly to
> Catalyst and compute them from stored input values after assignment casts.
> Share the conversion between auto-fill and validation. Prefer the modeled
> form, falling back to SQL only when the model is absent or unsupported by the
> converter. Report analysis and evaluation failures after conversion without
> changing representations.
> This changes results when both representations exist and differ, including
> implicit casts preserved in the modeled expression by CREATE TABLE. For
> example, coalesce(s, 1) defined under a non-ANSI session generates '007' for
> s = '007' from its modeled string fallback, where reparsing SQL previously
> produced '7'.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]