L. C. Hsieh created SPARK-61443:
-----------------------------------

             Summary: Fix the races of the progress handlers and streaming 
listeners, and the swallowed errors of the error conversion, of the Python 
client of Spark Connect Rust
                 Key: SPARK-61443
                 URL: https://issues.apache.org/jira/browse/SPARK-61443
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: L. C. Hsieh


The PyO3 extension of the Python client of spark-connect-rust 
(crates/pyspark-rs) declares gil_used = false, so two threads can be in the 
same object at once, and several of its paths also run Python code (__eq__, 
__del__) between two steps that were meant to be one.

* Progress handlers: the number of handlers that the client keeps beside the 
list is stored from a length read before other changes of the list. A handler 
registered while another change is in flight (another thread, or the __del__ of 
a handler that clearProgressHandlers drops) can be left uncounted, and is then 
never called.
* toLocalIterator: a second thread that calls next or close while a thread 
waits for a row gets "RuntimeError: Already borrowed". PySpark's generator 
raises "ValueError: generator already executing".
* Streaming listeners: addListener stores the ids of its registrations in a 
_rust_listener_ids attribute of the user's listener. That changes a user 
object, fails for an object that cannot take the attribute (__slots__), and 
loses registrations when several threads add and remove the same listener. 
PySpark's bus keeps its listeners in its own list and removes the first that is 
equal to the one given.
* Error conversion: a failure to make the exception of the connect classes, or 
of the class of the kind of the error, is dropped silently, as is a failure to 
set the __context__ of the done error of a progress handler or the 
_operation_id of an exception.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to