Hyukjin Kwon created SPARK-61492:
------------------------------------

             Summary: Upgrade PyO3 to 0.29 in the Python client, so that its 
PyO3 classes do not leak the instance dict that CPython 3.11 and 3.12 create
                 Key: SPARK-61492
                 URL: https://issues.apache.org/jira/browse/SPARK-61492
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


On CPython 3.11 and 3.12, object.__new__ creates the instance dict of every 
instance of a type with a dict offset (_PyObject_InitializeDict), and PyO3 0.28 
then overwrites that slot with null when it writes the contents of a 
#[pyclass(dict)] instance (pyclass_init.rs), so every instance of the Python 
client's dict classes (Column, DataFrame, SparkSession, DataType and its 
subclasses, GroupedData, readers and writers, Observation, ...) leaks an empty 
dict, 64 bytes, at creation, whether or not it ever gets an attribute: 20k 
Columns keep 1.28 MB. CPython 3.9 and 3.13 do not pre-create the dict. PyO3 
0.29 carries that dict over (pyclass_init.rs:161-185); it also releases the 
dict at dealloc (release_dict) and traverses it for the garbage collector 
(traverse_dict), which SPARK-61490 works around for the dealloc leak only.

The upgrade is blocked today: arrow-pyarrow 58.3.0 pins pyo3 0.28, and 
pyo3-ffi's links = "python" forbids two versions in one build. Upgrade PyO3 to 
0.29 together with an arrow/arrow-pyarrow release that supports it, then remove 
the tp_dealloc workaround of SPARK-61490 
(crates/pyspark-rs/src/instance_dict.rs), which stays harmless under 0.29 until 
removed.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to