L. C. Hsieh created SPARK-60933:
-----------------------------------
Summary: Make every DataType pickle round-trip in the Spark
Connect Rust drop-in
Key: SPARK-60933
URL: https://issues.apache.org/jira/browse/SPARK-60933
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
Fix For: connect-rust-4.3.0
In the Rust-backed PySpark drop-in, some DataTypes fail to pickle round-trip,
although every DataType must be picklable so it reaches an official-pyspark UDF
worker:
* GeometryType and GeographyType have no __reduce__, so
pickle.dumps(GeometryType(4326)) raises TypeError: cannot pickle
'pyspark.sql.types.GeometryType' object.
* The client-side pyspark.sql.types._parse_datatype_json_value does not parse
'string collate <collation>', so
pickle.loads(pickle.dumps(StringType('UTF8_LCASE'))) raises ValueError. It does
not parse the spatial JSON forms geometry(<crs>) / geography(<crs>, <alg>)
either.
* A bare unparsed DataType has no __reduce__, and the core JSON parser reads
'unparsed(x)' back as 'unparsed(unparsed(x))'.
The fix parses the JSON string forms with the Rust core's DataType::from_json
instead of a second parser in Python, adds __reduce__ to the spatial types and
a __reduce__ fallback to the base DataType, and makes 'unparsed(x)' parse back
to x.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
