[
https://issues.apache.org/jira/browse/SPARK-58525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-58525.
---------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 57733
[https://github.com/apache/spark/pull/57733]
> StructType.merge does not propagate the caseSensitive flag into nested
> struct/array/map field merges
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-58525
> URL: https://issues.apache.org/jira/browse/SPARK-58525
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Andreas Neumann
> Assignee: Andreas Neumann
> Priority: Major
> Labels: correctness, pull-request-available
> Fix For: 4.4.0
>
>
> h3. Summary
> {{StructType.merge(left, right, caseSensitive)}} honors the {{caseSensitive}}
> flag only for top-level struct fields. When it recurses into a matched
> field's type, it calls {{merge(leftType, rightType)}} without passing
> {{caseSensitive}}, so the parameter falls back to its {{= true}} default. As
> a result, a field whose name differs only in case, nested inside a matched
> struct / array<struct> / map<_, struct>, is treated as a distinct field even
> under case-insensitive merging -- producing a merged schema that carries both
> spellings (e.g. {{value}} and {{Value}}).
> h3. Root cause
> In {{sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala}},
> the struct-merge closure recurses via:
> {code:scala}
> leftField.copy(
> dataType = merge(leftType, rightType), // caseSensitive defaults back to
> true
> nullable = leftNullable || rightNullable)
> {code}
> The top-level comparison uses a case-aware {{normalize}} /
> {{fieldsMap(caseSensitive)}}, but the recursive call drops the flag.
> h3. History
> The {{caseSensitive}} parameter was added by SPARK-45346 ("Parquet schema
> inference should respect case sensitive flag when merging schema"), which
> fixed the top-level case but left the recursive call unchanged. The recursive
> merge itself dates back to the original implementation in SPARK-5182 (2015),
> which predates any case-sensitivity concept. This change completes
> SPARK-45346's intent for nested types.
> h3. Impact
> Affects every caller of the case-insensitive overload, not just one subsystem:
> * Parquet / ORC schema merging (via {{SchemaMergeUtils}}, which passes the
> session's {{caseSensitive}} to {{StructType.merge}}): nested
> case-only-differing fields were merged case-sensitively even under a
> case-insensitive session, so a later case-insensitive duplicate-column check
> could reject otherwise-mergeable files.
> * Any other caller relying on case-insensitive schema merge for nested types.
> h3. Fix
> Propagate {{caseSensitive}} through the recursive {{merge(leftType,
> rightType, caseSensitive)}} call so nested struct / array / map fields fold
> case-only differences onto the existing (left) field, consistent with the
> top-level behavior.
> h3. Tests
> Added coverage in {{StructTypeSuite}} for top-level and nested (struct,
> array<struct>, map<_, struct>) case-insensitive merges, plus the
> case-sensitive control that keeps both spellings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
