[ 
https://issues.apache.org/jira/browse/SPARK-61928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61928:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Do not read the state of the readers and writers that Python's garbage 
> collector traverses while they are created, which crashed the interpreter
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61928
>                 URL: https://issues.apache.org/jira/browse/SPARK-61928
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> On CPython 3.11, Python's garbage collector can traverse an object of the 
> drop-in's extension while the object is being created. PyO3 makes an object 
> of a #[pyclass(dict)] class, or of a Python subclass of it, with 
> object.__new__. That call tracks the object that it allocates, and then 
> allocates the object's __dict__. When the free list of dicts is empty, that 
> allocation can start a collection, before PyO3 writes the object's value. The 
> collector then calls the class's __traverse__ on zeroed memory.
> The __traverse__ of four classes read their state there:
> - DataFrameWriter and DataFrameWriterV2 read a null Arc (Arc::strong_count of 
> the kept arguments);
> - DataFrameReader and DataStreamReader read their zeroed Option of the unsent 
> format as Some of a null Py, and so gave the collector NULL.
> So creating one of them, for example df.write, spark.read, df.writeTo("t"), 
> spark.readStream or a copy of one, crashed the interpreter with SIGSEGV when 
> a collection started at that point. It is rare in normal use, as it needs a 
> collection that starts at that allocation, but a script whose threshold or 
> allocation pattern makes one start there crashes every time. For example, 
> gc.set_threshold(1) with the free list of dicts used up crashes `df.write` on 
> master. pyspark-client 4.3.0 has no such crash, as its classes are Python 
> classes.
> PyColumn and PyDataFrame already guarded their __traverse__ with a flag that 
> zeroed memory leaves false. The writers and readers should check such a flag 
> too, before they read anything else of the object.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to