Hyukjin Kwon created SPARK-61333:
------------------------------------

             Summary: Release pyspark.ml models whose summary keeps a DataFrame 
of them, as PySpark's garbage collector does
                 Key: SPARK-61333
                 URL: https://issues.apache.org/jira/browse/SPARK-61333
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


In the Python drop-in, a pyspark.ml KMeans, BisectingKMeans or GaussianMixture 
model is never released on the server: every fit leaks one server ML cache 
entry (offloaded to the driver's disk, and fits fail with 
MLCacheSizeOverflowException past the session's maxStorageSize) and its client 
objects. An elbow loop over k=2..5 leaves 4 models; official PySpark 4.3.0 
leaves none.
These models keep the default summary dataset, model.transform(dataset), whose 
plan keeps the model (__source_transformer__), a reference cycle that Python's 
garbage collector frees in PySpark, where the DataFrame is a Python object. In 
the drop-in, the DataFrame is a native object that keeps the Python plan in a 
Rust owner, which the collector cannot see, so the cycle is never freed. Other 
models (LogisticRegression, LinearSVC, ...) return the training dataset as 
their summary dataset and have no cycle.
The fix gives the native DataFrame garbage-collector support: it reports the 
Python plans that only it keeps alive (every reference on the way held once), 
and drops its plan when the collector breaks a cycle. A DataFrame that still 
uses the model keeps it, as in PySpark.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to