[ 
https://issues.apache.org/jira/browse/SPARK-61483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61483:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send the bytes of binary literal subclasses, corr and cov without a limit, 
> and check the Arrow schema of createDataFrame depth first, in the Python 
> client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61483
>                 URL: https://issues.apache.org/jira/browse/SPARK-61483
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Follow-ups of the review of SPARK-61410, SPARK-61414, SPARK-61434 and 
> SPARK-61435, each verified against PySpark 4.3.0:
> 1. lit() and the typed literals of the Python client (lit, isin, when, 
> otherwise, getItem, create_map, typed arrays, sql args) read the raw buffer 
> of a bytes or bytearray subclass, where PySpark sends bytes(value).
> 2. While raising an error of a client that the server-side session change 
> closed, the client still sent a Config to read the stack-trace setting; 
> PySpark sends nothing there.
> 3. corr and cov sent Limit(1) over their StatCorr/StatCov relation; PySpark 
> sends the relation as it is. The Python client's isEmpty skipped the 
> binaryAsBytes Config that PySpark's take -> collect reads after the execution.
> 4. createDataFrame of local rows checked the Arrow schema in two passes 
> instead of depth first, as PySpark's to_arrow_type does, so a schema with an 
> unsupported type before a non-nullable null type raised ValueError where 
> PySpark raises PySparkTypeError; and its "not a sequence" check accepted 
> generators, iterators, dict_keys, dict_items, frozensets and mappingproxies, 
> which PyArrow rejects with ArrowTypeError.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to