[
https://issues.apache.org/jira/browse/SPARK-61418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61418:
------------------------------------
Assignee: Hyukjin Kwon
> Make VariantVal, Geometry and Geography PySpark's plain classes in the Python
> client, whose attributes are instance attributes as given and which are freed
> with their values
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61418
> URL: https://issues.apache.org/jira/browse/SPARK-61418
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Python client's VariantVal, Geometry and Geography are PyO3 classes whose
> value/metadata and wkb/srid are Rust fields with read-only getters. PySpark's
> are plain Python classes whose __init__ stores its arguments, as given, as
> instance attributes. So, unlike PySpark's:
> - the attributes cannot be written or deleted, and a subclass whose __init__
> sets self.value (or self.wkb) fails: "attribute 'value' of
> 'pyspark.sql.types.VariantVal' objects is not writable";
> - vars(v) is {} where PySpark's is {'value': ..., 'metadata': ...};
> - VariantVal("x", None) or Geometry(bytearray(...), 4326) raise or convert
> where PySpark's store the objects as given.
> They are also #[pyclass(dict)] types, and PyO3 0.28 clears but never releases
> the __dict__ of such an instance when it is freed
> (PyClassDictSlot::clear_dict calls PyDict_Clear only; PyO3 0.29 adds
> release_dict). Since SPARK-61402, pickling a value creates its __dict__, so
> each pickled value, and each value with an attribute set, keeps 128 bytes
> after it is freed: 100k pickled VariantVals keep 12.8 MB (official PySpark:
> 88 bytes).
> Fix: embed PySpark 4.3.0's own three classes as Python source, as the client
> embeds Row (row.py), and build and read their attributes from Rust.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]