[
https://issues.apache.org/jira/browse/SPARK-61493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61493:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Ignore the prefetchPartitions of toLocalIterator, give its generator the code
> name toLocalIterator, and do not check the session IDs of artifact responses,
> as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61493
> URL: https://issues.apache.org/jira/browse/SPARK-61493
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Follow-ups of a post-hoc review of SPARK-61413 and SPARK-61410, checked
> against PySpark 4.3.0:
> - DataFrame.toLocalIterator of the Python client takes prefetchPartitions as
> a bool, so a value such as 1, None or "x" raises a TypeError at the call.
> PySpark Connect's toLocalIterator(self, prefetchPartitions: bool = False)
> ignores the value and returns the rows.
> - The generator that toLocalIterator returns has the code name _local_rows,
> which gi_code, tracebacks and inspect show. PySpark's generator has the code
> name toLocalIterator (qualified name DataFrame.toLocalIterator).
> - The Rust client checks the server-side session ID of the ArtifactStatus and
> AddArtifacts responses. So when spark.addArtifacts(...), or a cache of
> artifacts, is the first call after a server restart, it raises an assertion
> error and closes the client. PySpark's ArtifactManager does not check the IDs
> of these responses (and its requests carry no observed server-side session
> ID): the call returns, and the next call of the client raises.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]