Hyukjin Kwon created SPARK-61418:
------------------------------------
Summary: Make VariantVal, Geometry and Geography PySpark's plain
classes in the Python client, whose attributes are instance attributes as given
and which are freed with their values
Key: SPARK-61418
URL: https://issues.apache.org/jira/browse/SPARK-61418
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The Python client's VariantVal, Geometry and Geography are PyO3 classes whose
value/metadata and wkb/srid are Rust fields with read-only getters. PySpark's
are plain Python classes whose __init__ stores its arguments, as given, as
instance attributes. So, unlike PySpark's:
- the attributes cannot be written or deleted, and a subclass whose __init__
sets self.value (or self.wkb) fails: "attribute 'value' of
'pyspark.sql.types.VariantVal' objects is not writable";
- vars(v) is {} where PySpark's is {'value': ..., 'metadata': ...};
- VariantVal("x", None) or Geometry(bytearray(...), 4326) raise or convert
where PySpark's store the objects as given.
They are also #[pyclass(dict)] types, and PyO3 0.28 clears but never releases
the __dict__ of such an instance when it is freed (PyClassDictSlot::clear_dict
calls PyDict_Clear only; PyO3 0.29 adds release_dict). Since SPARK-61402,
pickling a value creates its __dict__, so each pickled value, and each value
with an attribute set, keeps 128 bytes after it is freed: 100k pickled
VariantVals keep 12.8 MB (official PySpark: 88 bytes).
Fix: embed PySpark 4.3.0's own three classes as Python source, as the client
embeds Row (row.py), and build and read their attributes from Rust.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]