[
https://issues.apache.org/jira/browse/SPARK-61335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61335:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send the fill values of fillna_value and fillna_map as PySpark sends them,
> and reject the others, in the Rust API
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61335
> URL: https://issues.apache.org/jira/browse/SPARK-61335
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> DataFrame::fillna_value and fillna_map of the Rust API sent a value of a type
> other than the ones PySpark's fillna takes as the string of its Debug text:
> Value::Null filled string columns with "Null", Value::Binary with "Binary([1,
> 2])", and List, Map, Struct and Time likewise. And Value::Integer and
> Value::Float, which PySpark's fillna(5) and fillna(1.5) send as a long and a
> double, were sent as an int and a float, which the server's fill rejects
> ("Unsupported value type java.lang.Integer (5).").
> As PySpark's fillna (NAFill._convert_value), the fix sends a bool as a
> boolean, an integer as a long, a float as a double and a string as a string,
> and rejects any other value with NOT_EXPECTED_TYPE and an empty map with
> CANNOT_BE_EMPTY: fillna_value and fillna_map now return a Result, as dropna
> does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]