Hyukjin Kwon created SPARK-61351:
------------------------------------

             Summary: Convert pandas frames with nullable or Arrow-backed 
column labels through Arrow, as PySpark does
                 Key: SPARK-61351
                 URL: https://issues.apache.org/jira/browse/SPARK-61351
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


A pandas frame whose column labels are an Index of pandas' nullable dtypes 
(Int64, Float64, boolean: a pivot, unstack or .T of convert_dtypes() data) or 
of Arrow-backed dtypes (int64[pyarrow], timestamp[pyarrow], date32[pyarrow], 
...), or a MultiIndex level or CategoricalIndex of them, took the native row 
path of createDataFrame in the Python drop-in, which converts cells differently 
from PySpark 4.3.0, which converts the frame through Arrow. For example, an 
object column [1, 2.5] failed with "Type mismatch in row data" (PySpark: 
double), a dict cell gave a map (PySpark: struct), a NaN in a list stayed NaN 
(PySpark: None), np.int32 gave bigint (PySpark: int), and under 
spark.sql.session.timeZone=Asia/Seoul with a client in Asia/Seoul, a datetime64 
column under an Int64 label with an explicit timestamp_ntz schema read back 
2020-01-01 18:00 (PySpark: 2020-01-02 03:00).
SPARK-61314 fixed the same for DatetimeIndex, TimedeltaIndex, PeriodIndex and 
IntervalIndex labels. The fix accepts the exact IntegerArray, FloatingArray, 
BooleanArray and ArrowExtensionArray storages of a label Index on the Arrow 
routes, with the same checks as for the string labels: pandas' own classes and 
hooks, shadowed by no instance attribute, in pandas 2 and 3.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to