[
https://issues.apache.org/jira/browse/SPARK-60954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60954:
------------------------------------
Assignee: Hyukjin Kwon
> Type and convert all-null, Decimal and map columns of pandas input in Spark
> Connect Rust createDataFrame as PySpark does
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60954
> URL: https://issues.apache.org/jira/browse/SPARK-60954
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect converts a pandas DataFrame through Arrow, which types each
> column from its pandas dtype and values. The Rust-backed Python client types
> each column from its first non-null value, so these PySpark 4.3.0 results
> differ:
> * A column whose values are all NaN, NaT or None, such as
> {{pd.DataFrame({"id": [1, 2], "s": [np.nan, np.nan]})}}, gives {{double}}
> (and {{void}} for an object column of None). The client raises
> {{CANNOT_DETERMINE_TYPE}}.
> * A column of {{Decimal}} values, such as {{Decimal("1.5")}} and
> {{Decimal("2.25")}}, gives {{decimal(3,2)}} with the values {{1.50}} and
> {{2.25}}. The client gives {{decimal(38,1)}} from the scale of the first
> value, and silently truncates {{2.25}} to {{2.2}}.
> * A column of dicts with a {{None}} key raises a {{PySparkTypeError}} of the
> Arrow conversion. The client panics in Rust ({{PanicException}}), as does the
> Rust {{SparkSession::create_dataframe}} with a null map key.
> Type and convert these columns as PySpark does, and raise an error for a null
> map key.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
