[
https://issues.apache.org/jira/browse/SPARK-59268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59268:
-----------------------------------
Labels: pull-request-available (was: )
> [SDP] Preserve desired column metadata during additive schema evolution
> -----------------------------------------------------------------------
>
> Key: SPARK-59268
> URL: https://issues.apache.org/jira/browse/SPARK-59268
> Project: Spark
> Issue Type: Bug
> Components: Declarative Pipelines
> Affects Versions: 4.2.0
> Reporter: Anish Mahto
> Priority: Minor
> Labels: pull-request-available
>
> During incremental dataset evolution, SDP computes the target schema using:
> ```
> SchemaMergingUtils.mergeSchemas(currentSchema, desiredSchema, caseSensitive)
> ```
> Today, `StructType.merge` retains the left/current field metadata and comment
> for matching columns. Consequently, changes made in the desired schema such
> as modifying, adding, or dropping an existing column’s comment or current
> default value, are discarded before `diffSchemas` runs to actually reconcile
> the changing schema attributes.
> Either `StructType.merge` should update the left field's metadata and comment
> sub-fields the same way it already does for `dataType` and `nullable`, SDP
> should implement its own more comprehensive merge, or SDP should explicitly
> reject changes to an existing column's comment or default value on schema
> evolution.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]