Hyukjin Kwon created SPARK-61493:
------------------------------------
Summary: Ignore the prefetchPartitions of toLocalIterator, give
its generator the code name toLocalIterator, and do not check the session IDs
of artifact responses, as PySpark does
Key: SPARK-61493
URL: https://issues.apache.org/jira/browse/SPARK-61493
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Follow-ups of a post-hoc review of SPARK-61413 and SPARK-61410, checked against
PySpark 4.3.0:
- DataFrame.toLocalIterator of the Python client takes prefetchPartitions as a
bool, so a value such as 1, None or "x" raises a TypeError at the call. PySpark
Connect's toLocalIterator(self, prefetchPartitions: bool = False) ignores the
value and returns the rows.
- The generator that toLocalIterator returns has the code name _local_rows,
which gi_code, tracebacks and inspect show. PySpark's generator has the code
name toLocalIterator (qualified name DataFrame.toLocalIterator).
- The Rust client checks the server-side session ID of the ArtifactStatus and
AddArtifacts responses. So when spark.addArtifacts(...), or a cache of
artifacts, is the first call after a server restart, it raises an assertion
error and closes the client. PySpark's ArtifactManager does not check the IDs
of these responses (and its requests carry no observed server-side session ID):
the call returns, and the next call of the client raises.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]