[ 
https://issues.apache.org/jira/browse/SPARK-62017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62017:
------------------------------------

    Assignee: Hyukjin Kwon

> Report the types, flags, metadata and fields that the array, map, field and 
> struct types hold to Python's garbage collector
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62017
>                 URL: https://issues.apache.org/jira/browse/SPARK-62017
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> PySpark's ArrayType, MapType, StructField and StructType are plain Python 
> classes. Their __dict__
> holds the type objects that they are made of, their flags, the metadata dict 
> of a field, and the
> fields and names lists of a struct, so the garbage collector collects a 
> reference cycle through
> them. The drop-in holds them in Rust, and the four classes have no 
> __traverse__. So such a cycle is
> never collected, as SPARK-61971 found of the other holders of Python objects:
> - an array of a DecimalType that has an attribute that keeps the array;
> - a map of such a key or value;
> - a field of such a type, or of a metadata dict that keeps the field;
> - a struct of such a field, also of StructType.add.
> Repro (master 50594b2e)
>   >>> class Job: pass
>   >>> job = Job(); t = DecimalType(10, 2); t.job = job
>   >>> job.held = ArrayType(t)
>   >>> ref = weakref.ref(job); del job, t; gc.collect(); ref() is None
>   False
> Expected (as with pyspark-client 4.3.0)
>   True, and so for the map, the field, the metadata dict and the struct. A 
> child that is a
>   singleton, such as LongType(), keeps the cycle in both clients, as 
> PySpark's singleton holds it.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to