Hyukjin Kwon created SPARK-60908:
------------------------------------
Summary: Infer createDataFrame types in Spark Connect Rust by
merging all rows as PySpark does
Key: SPARK-60908
URL: https://issues.apache.org/jira/browse/SPARK-60908
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect infers the schema of local rows in {{createDataFrame}} by
inferring each row's type and merging them ({{_infer_type}} and
{{_merge_type}}). The Rust-backed Python client takes each top-level field's
type from its first non-null value, and a nested type from the first non-null
element or map value, so these PySpark 4.3.0 results differ:
* Mixed atomic types and strings merge into a string type, and the other values
become strings: {{createDataFrame([(1,), ("a",)])}} gives a string column with
{{'1'}} and {{'a'}}, and {{[([1, "a"],)]}} gives {{array<string>}}. The client
raises an unclassified error.
* Other conflicts raise {{CANNOT_MERGE_TYPE}} (for example {{[(1,), (2.5,)]}}),
or {{CANNOT_INFER_TYPE_FOR_FIELD}} within a field (for example {{[([1,
2.5],)]}}). The client raises an unclassified error.
* Nested values that are null in every row raise {{CANNOT_DETERMINE_TYPE}}, as
top-level fields do: {{[([None],)]}} and {{[({"k": None},)]}}. The client
infers {{array<string>}} and {{map<string,string>}}. A nested type that only a
later row determines, as in {{[([None],), ([1],)]}}, gives {{array<bigint>}};
the client raises an error.
* A {{Decimal}} value infers {{decimal(38,18)}}; the client infers
{{decimal(38,s)}}, with the value's scale.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
