[
https://issues.apache.org/jira/browse/SPARK-61443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61443:
-----------------------------------
Labels: pull-request-available (was: )
> Fix the races of the progress handlers and streaming listeners, and the
> swallowed errors of the error conversion, of the Python client of Spark
> Connect Rust
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61443
> URL: https://issues.apache.org/jira/browse/SPARK-61443
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The PyO3 extension of the Python client of spark-connect-rust
> (crates/pyspark-rs) declares gil_used = false, so two threads can be in the
> same object at once, and several of its paths also run Python code (__eq__,
> __del__) between two steps that were meant to be one.
> * Progress handlers: the number of handlers that the client keeps beside the
> list is stored from a length read before other changes of the list. A handler
> registered while another change is in flight (another thread, or the __del__
> of a handler that clearProgressHandlers drops) can be left uncounted, and is
> then never called.
> * toLocalIterator: a second thread that calls next or close while a thread
> waits for a row gets "RuntimeError: Already borrowed". PySpark's generator
> raises "ValueError: generator already executing".
> * Streaming listeners: addListener stores the ids of its registrations in a
> _rust_listener_ids attribute of the user's listener. That changes a user
> object, fails for an object that cannot take the attribute (__slots__), and
> loses registrations when several threads add and remove the same listener.
> PySpark's bus keeps its listeners in its own list and removes the first that
> is equal to the one given.
> * Error conversion: a failure to make the exception of the connect classes,
> or of the class of the kind of the error, is dropped silently, as is a
> failure to set the __context__ of the done error of a progress handler or the
> _operation_id of an exception.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]