[ 
https://issues.apache.org/jira/browse/SPARK-61492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61492.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 579
[https://github.com/apache/spark-connect-rust/pull/579]

> Upgrade PyO3 to 0.29 in the Python client, so that its PyO3 classes do not 
> leak the instance dict that CPython 3.11 and 3.12 create
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61492
>                 URL: https://issues.apache.org/jira/browse/SPARK-61492
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> On CPython 3.11 and 3.12, object.__new__ creates the instance dict of every 
> instance of a type with a dict offset (_PyObject_InitializeDict), and PyO3 
> 0.28 then overwrites that slot with null when it writes the contents of a 
> #[pyclass(dict)] instance (pyclass_init.rs), so every instance of the Python 
> client's dict classes (Column, DataFrame, SparkSession, DataType and its 
> subclasses, GroupedData, readers and writers, Observation, ...) leaks an 
> empty dict, 64 bytes, at creation, whether or not it ever gets an attribute: 
> 20k Columns keep 1.28 MB. CPython 3.9 and 3.13 do not pre-create the dict. 
> PyO3 0.29 carries that dict over (pyclass_init.rs:161-185); it also releases 
> the dict at dealloc (release_dict) and traverses it for the garbage collector 
> (traverse_dict), which SPARK-61490 works around for the dealloc leak only.
> The upgrade is blocked today: arrow-pyarrow 58.3.0 pins pyo3 0.28, and 
> pyo3-ffi's links = "python" forbids two versions in one build. Upgrade PyO3 
> to 0.29 together with an arrow/arrow-pyarrow release that supports it, then 
> remove the tp_dealloc workaround of SPARK-61490 
> (crates/pyspark-rs/src/instance_dict.rs), which stays harmless under 0.29 
> until removed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to