Hyukjin Kwon created SPARK-61490:
------------------------------------

             Summary: Release the instance dict of the Python client's PyO3 
classes when they are freed, which PyO3 0.28 keeps
                 Key: SPARK-61490
                 URL: https://issues.apache.org/jira/browse/SPARK-61490
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


PyO3 0.28 never releases the __dict__ of a #[pyclass(dict)] instance: 
PyClassObjectContents::dealloc calls clear_dict, which is PyDict_Clear, and 
does not decref the dict (PyO3 0.29 adds release_dict). Every instance of the 
Python client's 28 dict classes (Column, DataFrame, SparkSession, DataType and 
its Rust subclasses, GroupedData, readers and writers, Observation, ...) whose 
__dict__ was created (an attribute set, vars(), dir(), monkeypatching, a 
__reduce__ that reads it) keeps an empty dict, 64 bytes, after it is freed. And 
since PyDict_Clear clears outside CPython's trashcan, freeing a long chain of 
instances linked through their attributes (100k Columns) overflows the stack 
and crashes the process.

PyO3 cannot be upgraded here: arrow-pyarrow 58.3.0 pins pyo3 0.28.

Fix: wrap the tp_dealloc of each dict class, and of the Rust subclasses of 
DataType, so that the instance dict is taken out of its slot before PyO3's 
dealloc and released after it, only where the type object holds the tp_dealloc 
that PyType_GetSlot reports. Note: on CPython 3.11 and 3.12, object.__new__ 
also creates a dict for every instance of a type with a dict offset, which PyO3 
0.28 overwrites; that creation leak needs PyO3 0.29.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to