[ 
https://issues.apache.org/jira/browse/SPARK-61418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61418.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 493
[https://github.com/apache/spark-connect-rust/pull/493]

> Make VariantVal, Geometry and Geography PySpark's plain classes in the Python 
> client, whose attributes are instance attributes as given and which are freed 
> with their values
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61418
>                 URL: https://issues.apache.org/jira/browse/SPARK-61418
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> The Python client's VariantVal, Geometry and Geography are PyO3 classes whose 
> value/metadata and wkb/srid are Rust fields with read-only getters. PySpark's 
> are plain Python classes whose __init__ stores its arguments, as given, as 
> instance attributes. So, unlike PySpark's:
> - the attributes cannot be written or deleted, and a subclass whose __init__ 
> sets self.value (or self.wkb) fails: "attribute 'value' of 
> 'pyspark.sql.types.VariantVal' objects is not writable";
> - vars(v) is {} where PySpark's is {'value': ..., 'metadata': ...};
> - VariantVal("x", None) or Geometry(bytearray(...), 4326) raise or convert 
> where PySpark's store the objects as given.
> They are also #[pyclass(dict)] types, and PyO3 0.28 clears but never releases 
> the __dict__ of such an instance when it is freed 
> (PyClassDictSlot::clear_dict calls PyDict_Clear only; PyO3 0.29 adds 
> release_dict). Since SPARK-61402, pickling a value creates its __dict__, so 
> each pickled value, and each value with an attribute set, keeps 128 bytes 
> after it is freed: 100k pickled VariantVals keep 12.8 MB (official PySpark: 
> 88 bytes).
> Fix: embed PySpark 4.3.0's own three classes as Python source, as the client 
> embeds Row (row.py), and build and read their attributes from Rust.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to