[
https://issues.apache.org/jira/browse/SPARK-60908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60908:
------------------------------------
Assignee: Hyukjin Kwon
> Infer createDataFrame types in Spark Connect Rust by merging all rows as
> PySpark does
> -------------------------------------------------------------------------------------
>
> Key: SPARK-60908
> URL: https://issues.apache.org/jira/browse/SPARK-60908
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect infers the schema of local rows in {{createDataFrame}} by
> inferring each row's type and merging them ({{_infer_type}} and
> {{_merge_type}}). The Rust-backed Python client takes each top-level field's
> type from its first non-null value, and a nested type from the first non-null
> element or map value, so these PySpark 4.3.0 results differ:
> * Mixed atomic types and strings merge into a string type, and the other
> values become strings: {{createDataFrame([(1,), ("a",)])}} gives a string
> column with {{'1'}} and {{'a'}}, and {{[([1, "a"],)]}} gives
> {{array<string>}}. The client raises an unclassified error.
> * Other conflicts raise {{CANNOT_MERGE_TYPE}} (for example {{[(1,),
> (2.5,)]}}), or {{CANNOT_INFER_TYPE_FOR_FIELD}} within a field (for example
> {{[([1, 2.5],)]}}). The client raises an unclassified error.
> * Nested values that are null in every row raise {{CANNOT_DETERMINE_TYPE}},
> as top-level fields do: {{[([None],)]}} and {{[({"k": None},)]}}. The client
> infers {{array<string>}} and {{map<string,string>}}. A nested type that only
> a later row determines, as in {{[([None],), ([1],)]}}, gives
> {{array<bigint>}}; the client raises an error.
> * A {{Decimal}} value infers {{decimal(38,18)}}; the client infers
> {{decimal(38,s)}}, with the value's scale.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
