[
https://issues.apache.org/jira/browse/SPARK-58347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho resolved SPARK-58347.
-------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57610
[https://github.com/apache/spark/pull/57610]
> Thread conf.resolver through ColumnSelection.applyToSchema instead of a
> caseSensitive boolean
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-58347
> URL: https://issues.apache.org/jira/browse/SPARK-58347
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 5.0.0
> Reporter: Andreas Neumann
> Assignee: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Follow-up from review of SPARK-58313 (PR #57490). AutoCDC column-name
> resolution is currently driven by a caseSensitive: Boolean that each call
> site derives from conf.caseSensitiveAnalysis, then re-derives a resolver (or
> uses StructType.getFieldIndex / getFieldIndexCaseInsensitive) internally. A
> reviewer suggested threading conf.resolver all the way through for
> futureproofing, so resolution semantics live in one place rather than being
> reconstructed from a boolean at each layer.
> The blocker for doing this within SPARK-58313 was scope:
> ColumnSelection.applyToSchema (and its helper lookupFieldIndices) take a
> caseSensitive: Boolean and rely on StructType.getFieldIndex /
> getFieldIndexCaseInsensitive, which are boolean-driven. Threading a Resolver
> requires reworking applyToSchema/lookupFieldIndices to be resolver-based and
> updating all callers.
> Scope:
> - Rework ColumnSelection.applyToSchema and lookupFieldIndices to accept/use a
> Resolver (or a small resolution abstraction) instead of a Boolean.
> - Update all callers to pass conf.resolver: Scd1BatchProcessor (2 sites),
> Scd2BatchProcessor (preprocess projection + computeTrackedHistoryColumns),
> and Flow (userSelectedSchema).
> - Scd2BatchProcessor.computeTrackedHistoryColumns can then take a Resolver
> directly instead of deriving one from a caseSensitive boolean.
> This is a pure refactor with no behavior change (case-sensitive and
> case-insensitive resolution must remain identical); it also touches the
> shared SCD1 code paths, which is why it is intentionally deferred out of
> SPARK-58313.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
