[
https://issues.apache.org/jira/browse/SPARK-61483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61483:
-----------------------------------
Labels: pull-request-available (was: )
> Send the bytes of binary literal subclasses, corr and cov without a limit,
> and check the Arrow schema of createDataFrame depth first, in the Python
> client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61483
> URL: https://issues.apache.org/jira/browse/SPARK-61483
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Follow-ups of the review of SPARK-61410, SPARK-61414, SPARK-61434 and
> SPARK-61435, each verified against PySpark 4.3.0:
> 1. lit() and the typed literals of the Python client (lit, isin, when,
> otherwise, getItem, create_map, typed arrays, sql args) read the raw buffer
> of a bytes or bytearray subclass, where PySpark sends bytes(value).
> 2. While raising an error of a client that the server-side session change
> closed, the client still sent a Config to read the stack-trace setting;
> PySpark sends nothing there.
> 3. corr and cov sent Limit(1) over their StatCorr/StatCov relation; PySpark
> sends the relation as it is. The Python client's isEmpty skipped the
> binaryAsBytes Config that PySpark's take -> collect reads after the execution.
> 4. createDataFrame of local rows checked the Arrow schema in two passes
> instead of depth first, as PySpark's to_arrow_type does, so a schema with an
> unsupported type before a non-nullable null type raised ValueError where
> PySpark raises PySparkTypeError; and its "not a sequence" check accepted
> generators, iterators, dict_keys, dict_items, frozensets and mappingproxies,
> which PyArrow rejects with ArrowTypeError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]