Hyukjin Kwon created SPARK-61849:
------------------------------------
Summary: Load the data types that PySpark pickled, and make the
parameters of a type its attributes as PySpark's
Key: SPARK-61849
URL: https://issues.apache.org/jira/browse/SPARK-61849
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
PySpark's data types are plain classes, which pickle as their class and their
__dict__ (copyreg.__newobj__ at protocols 2 and up). The drop-in's native type
classes require their constructor arguments and have no __setstate__, so the
pickle of a type that official PySpark made does not load right in the drop-in:
- DecimalType(10,2) loads as DecimalType(10,0), StringType('UTF8_LCASE') as
StringType(), TimeType(3) as TimeType(6), DayTimeIntervalType(1,2) as
DayTimeIntervalType(0,3);
- ArrayType, MapType, StructField, StructType, CharType, VarcharType,
GeometryType and GeographyType raise TypeError ("missing 1 required positional
argument"), so a pickled df.schema does not load at all.
Also, the parameters of a type (DecimalType.scale, StringType.collation,
CharType.length, TimeType.precision, the interval fields, ...) are plain
attributes in PySpark, whose assignment changes the type; in the drop-in they
are read-only.
Make PySpark's pickles of the types load as the same types at protocols 2 to 5,
and the parameters assignable as PySpark's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]