Hyukjin Kwon created SPARK-61490:
------------------------------------
Summary: Release the instance dict of the Python client's PyO3
classes when they are freed, which PyO3 0.28 keeps
Key: SPARK-61490
URL: https://issues.apache.org/jira/browse/SPARK-61490
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PyO3 0.28 never releases the __dict__ of a #[pyclass(dict)] instance:
PyClassObjectContents::dealloc calls clear_dict, which is PyDict_Clear, and
does not decref the dict (PyO3 0.29 adds release_dict). Every instance of the
Python client's 28 dict classes (Column, DataFrame, SparkSession, DataType and
its Rust subclasses, GroupedData, readers and writers, Observation, ...) whose
__dict__ was created (an attribute set, vars(), dir(), monkeypatching, a
__reduce__ that reads it) keeps an empty dict, 64 bytes, after it is freed. And
since PyDict_Clear clears outside CPython's trashcan, freeing a long chain of
instances linked through their attributes (100k Columns) overflows the stack
and crashes the process.
PyO3 cannot be upgraded here: arrow-pyarrow 58.3.0 pins pyo3 0.28.
Fix: wrap the tp_dealloc of each dict class, and of the Rust subclasses of
DataType, so that the instance dict is taken out of its slot before PyO3's
dealloc and released after it, only where the type object holds the tp_dealloc
that PyType_GetSlot reports. Note: on CPython 3.11 and 3.12, object.__new__
also creates a dict for every instance of a type with a dict offset, which PyO3
0.28 overwrites; that creation leak needs PyO3 0.29.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]