Hyukjin Kwon created SPARK-61335:
------------------------------------

             Summary: Send the fill values of fillna_value and fillna_map as 
PySpark sends them, and reject the others, in the Rust API
                 Key: SPARK-61335
                 URL: https://issues.apache.org/jira/browse/SPARK-61335
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


DataFrame::fillna_value and fillna_map of the Rust API sent a value of a type 
other than the ones PySpark's fillna takes as the string of its Debug text: 
Value::Null filled string columns with "Null", Value::Binary with "Binary([1, 
2])", and List, Map, Struct and Time likewise. And Value::Integer and 
Value::Float, which PySpark's fillna(5) and fillna(1.5) send as a long and a 
double, were sent as an int and a float, which the server's fill rejects 
("Unsupported value type java.lang.Integer (5).").
As PySpark's fillna (NAFill._convert_value), the fix sends a bool as a boolean, 
an integer as a long, a float as a double and a string as a string, and rejects 
any other value with NOT_EXPECTED_TYPE and an empty map with CANNOT_BE_EMPTY: 
fillna_value and fillna_map now return a Result, as dropna does.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to