[ 
https://issues.apache.org/jira/browse/SPARK-62065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62065:
-----------------------------------
    Labels: pull-request-available  (was: )

> [Connect][Python] Drop-in: hold PySpark's user-defined type objects in the 
> cached schemas of set operations and pickles, raise the conversion errors of 
> the schema of the server where PySpark reads it, and make 
> DataType.simpleString the typeName
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62065
>                 URL: https://issues.apache.org/jira/browse/SPARK-62065
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The drop-in's cached schema of a DataFrame, which a set operation compares 
> and a pickle carries, and some reads of the schema of the server do not hold 
> or convert the user-defined type (UDT) objects as pyspark-client 4.3.0 does.
> 1. Set operations. PySpark's _merge_cached_schema keeps the cached schema of 
> `this` when it equals that of `other` by ==, which compares the __dict__ of a 
> UDT (python/pyspark/sql/connect/dataframe.py:1203-1208). The cached schema of 
> a DataFrame of the schema of the server is of new UDT objects 
> (UserDefinedType.fromJson, python/pyspark/sql/types.py:1986-2004). The 
> drop-in compared it with the UDT object that it had lowered last of its JSON, 
> often the user's own.
>     u = PtUDT(); u.marker = "made"
>     server = spark.createDataFrame([((1.0, 2.0),)], 
> StructType([StructField("p", PtUDT())]))
>     server.columns
>     cast = server.to(StructType([StructField("p", u)]))
>     hasattr(cast.union(server).schema["p"].dataType, "marker")
>     # PySpark: False (the cached schemas differ, the union caches none, its 
> schema is the server's)
>     # drop-in: True (the deep copy of the schema of `to`)
> 2. Pickles. PySpark's __reduce__ carries _cached_schema as its objects 
> (dataframe.py:150-170), which the unpickled DataFrame keeps: dtypes, columns 
> and the methods that copy the cached schema read them, while schema is the 
> one that its unpickle requested. The drop-in lowered the pickled StructType 
> into a core type and dropped it, and pickled the registry's objects of a 
> DataFrame of the server.
>     
> cloudpickle.loads(cloudpickle.dumps(cast)).filter("true").schema["p"].dataType.marker
>     # PySpark: "made"; drop-in: AttributeError (a new object)
> 3. The reads of the schema. PySpark's _schema converts the schema of the 
> server (dataframe.py:2024-2038, python/pyspark/sql/connect/types.py:320-327), 
> so every read of it raises the errors of that conversion: the TypeError of a 
> UDT whose __init__ needs arguments, the KeyError of a UDT of no Python class, 
> UNSUPPORTED_OPERATION of a nanosecond timestamp. The drop-in raised them for 
> schema, dtypes and the rows, but not for columns, printSchema, df[0], 
> dir(df), a lookup of a name of two underscores, the columns of mapInPandas 
> and applyInPandas, copy.copy, copy.deepcopy, an unpickle, or 
> GroupedData.sum() and the other numeric aggregations, which read 
> self._df.schema (python/pyspark/sql/connect/group.py:160).
>     class ArgUDT(PtUDT):
>         def __init__(self, scale): self.scale = scale
>     df = spark.createDataFrame([((1.0, 2.0), 1)], 
> StructType([StructField("p", ArgUDT(2)), StructField("i", LongType())]))
>     df.columns            # PySpark: TypeError: ArgUDT.__init__() missing 1 
> required positional argument: 'scale'; drop-in: ['p', 'i']
>     copy.copy(df)         # PySpark: the same TypeError; drop-in: a DataFrame
>     df.groupBy("i").sum() # PySpark: the same TypeError; drop-in: a DataFrame
> 4. DataType.simpleString. PySpark's base method is self.typeName() 
> (types.py:134-135), which the types of a string of their own override, so 
> DataType.simpleString(StructType([...])) is "struct" and 
> DataType.simpleString(LongType()) is "long". The drop-in's gave the simple 
> string of the type ("struct<a:bigint>", "bigint"), of the registry's UDT 
> objects for a struct, array or map, and did not read the typeName() of a 
> subclass.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to