[
https://issues.apache.org/jira/browse/SPARK-62065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62065:
-----------------------------------
Labels: pull-request-available (was: )
> [Connect][Python] Drop-in: hold PySpark's user-defined type objects in the
> cached schemas of set operations and pickles, raise the conversion errors of
> the schema of the server where PySpark reads it, and make
> DataType.simpleString the typeName
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62065
> URL: https://issues.apache.org/jira/browse/SPARK-62065
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> The drop-in's cached schema of a DataFrame, which a set operation compares
> and a pickle carries, and some reads of the schema of the server do not hold
> or convert the user-defined type (UDT) objects as pyspark-client 4.3.0 does.
> 1. Set operations. PySpark's _merge_cached_schema keeps the cached schema of
> `this` when it equals that of `other` by ==, which compares the __dict__ of a
> UDT (python/pyspark/sql/connect/dataframe.py:1203-1208). The cached schema of
> a DataFrame of the schema of the server is of new UDT objects
> (UserDefinedType.fromJson, python/pyspark/sql/types.py:1986-2004). The
> drop-in compared it with the UDT object that it had lowered last of its JSON,
> often the user's own.
> u = PtUDT(); u.marker = "made"
> server = spark.createDataFrame([((1.0, 2.0),)],
> StructType([StructField("p", PtUDT())]))
> server.columns
> cast = server.to(StructType([StructField("p", u)]))
> hasattr(cast.union(server).schema["p"].dataType, "marker")
> # PySpark: False (the cached schemas differ, the union caches none, its
> schema is the server's)
> # drop-in: True (the deep copy of the schema of `to`)
> 2. Pickles. PySpark's __reduce__ carries _cached_schema as its objects
> (dataframe.py:150-170), which the unpickled DataFrame keeps: dtypes, columns
> and the methods that copy the cached schema read them, while schema is the
> one that its unpickle requested. The drop-in lowered the pickled StructType
> into a core type and dropped it, and pickled the registry's objects of a
> DataFrame of the server.
>
> cloudpickle.loads(cloudpickle.dumps(cast)).filter("true").schema["p"].dataType.marker
> # PySpark: "made"; drop-in: AttributeError (a new object)
> 3. The reads of the schema. PySpark's _schema converts the schema of the
> server (dataframe.py:2024-2038, python/pyspark/sql/connect/types.py:320-327),
> so every read of it raises the errors of that conversion: the TypeError of a
> UDT whose __init__ needs arguments, the KeyError of a UDT of no Python class,
> UNSUPPORTED_OPERATION of a nanosecond timestamp. The drop-in raised them for
> schema, dtypes and the rows, but not for columns, printSchema, df[0],
> dir(df), a lookup of a name of two underscores, the columns of mapInPandas
> and applyInPandas, copy.copy, copy.deepcopy, an unpickle, or
> GroupedData.sum() and the other numeric aggregations, which read
> self._df.schema (python/pyspark/sql/connect/group.py:160).
> class ArgUDT(PtUDT):
> def __init__(self, scale): self.scale = scale
> df = spark.createDataFrame([((1.0, 2.0), 1)],
> StructType([StructField("p", ArgUDT(2)), StructField("i", LongType())]))
> df.columns # PySpark: TypeError: ArgUDT.__init__() missing 1
> required positional argument: 'scale'; drop-in: ['p', 'i']
> copy.copy(df) # PySpark: the same TypeError; drop-in: a DataFrame
> df.groupBy("i").sum() # PySpark: the same TypeError; drop-in: a DataFrame
> 4. DataType.simpleString. PySpark's base method is self.typeName()
> (types.py:134-135), which the types of a string of their own override, so
> DataType.simpleString(StructType([...])) is "struct" and
> DataType.simpleString(LongType()) is "long". The drop-in's gave the simple
> string of the type ("struct<a:bigint>", "bigint"), of the registry's UDT
> objects for a struct, array or map, and did not read the typeName() of a
> subclass.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]