Hyukjin Kwon created SPARK-61483:
------------------------------------
Summary: Send the bytes of binary literal subclasses, corr and cov
without a limit, and check the Arrow schema of createDataFrame depth first, in
the Python client as PySpark does
Key: SPARK-61483
URL: https://issues.apache.org/jira/browse/SPARK-61483
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Follow-ups of the review of SPARK-61410, SPARK-61414, SPARK-61434 and
SPARK-61435, each verified against PySpark 4.3.0:
1. lit() and the typed literals of the Python client (lit, isin, when,
otherwise, getItem, create_map, typed arrays, sql args) read the raw buffer of
a bytes or bytearray subclass, where PySpark sends bytes(value).
2. While raising an error of a client that the server-side session change
closed, the client still sent a Config to read the stack-trace setting; PySpark
sends nothing there.
3. corr and cov sent Limit(1) over their StatCorr/StatCov relation; PySpark
sends the relation as it is. The Python client's isEmpty skipped the
binaryAsBytes Config that PySpark's take -> collect reads after the execution.
4. createDataFrame of local rows checked the Arrow schema in two passes instead
of depth first, as PySpark's to_arrow_type does, so a schema with an
unsupported type before a non-nullable null type raised ValueError where
PySpark raises PySparkTypeError; and its "not a sequence" check accepted
generators, iterators, dict_keys, dict_items, frozensets and mappingproxies,
which PyArrow rejects with ArrowTypeError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]