[
https://issues.apache.org/jira/browse/SPARK-60933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60933:
-----------------------------------
Labels: pull-request-available (was: )
> Make every DataType pickle round-trip in the Spark Connect Rust drop-in
> -----------------------------------------------------------------------
>
> Key: SPARK-60933
> URL: https://issues.apache.org/jira/browse/SPARK-60933
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In the Rust-backed PySpark drop-in, some DataTypes fail to pickle round-trip,
> although every DataType must be picklable so it reaches an official-pyspark
> UDF worker:
> * GeometryType and GeographyType have no __reduce__, so
> pickle.dumps(GeometryType(4326)) raises TypeError: cannot pickle
> 'pyspark.sql.types.GeometryType' object.
> * The client-side pyspark.sql.types._parse_datatype_json_value does not parse
> 'string collate <collation>', so
> pickle.loads(pickle.dumps(StringType('UTF8_LCASE'))) raises ValueError. It
> does not parse the spatial JSON forms geometry(<crs>) / geography(<crs>,
> <alg>) either.
> * A bare unparsed DataType has no __reduce__, and the core JSON parser reads
> 'unparsed(x)' back as 'unparsed(unparsed(x))'.
> The fix parses the JSON string forms with the Rust core's DataType::from_json
> instead of a second parser in Python, adds __reduce__ to the spatial types
> and a __reduce__ fallback to the base DataType, and makes 'unparsed(x)' parse
> back to x.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
