[ 
https://issues.apache.org/jira/browse/SPARK-62016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62016:
------------------------------------

    Assignee: Hyukjin Kwon

> Treat user-defined type objects as PySpark does: keep them in the types that 
> hold them until a use needs their JSON, make new ones for the schemas of the 
> server, and render casts by their simpleString()
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62016
>                 URL: https://issues.apache.org/jira/browse/SPARK-62016
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> The drop-in differs from pyspark-client 4.3.0 in four ways on user-defined 
> type (UDT) objects:
> 1. The ArrayType, MapType, StructField and StructType of a UDT whose module() 
> or sqlType() is not
>    implemented, such as UserDefinedType() itself, raise when they are made. 
> PySpark keeps the type
>    object as it is and reads it only for a use that needs its JSON or its 
> conversion. So in
>    PySpark ArrayType(UserDefinedType()) is made, its repr, simpleString, ==, 
> hash, pickles and
>    tree string work, and only json(), a plan or toInternal raise.
> 2. The schema of a DataFrame holds the UDT object that the client last 
> lowered of its JSON: the
>    user's own object, with its attributes. Lowering an unrelated UDT of the 
> same JSON later changes
>    the schema of an untouched DataFrame. PySpark's UserDefinedType.fromJson 
> makes a new UDT() of the
>    class (python/pyspark/sql/types.py:1986-2005), which raises TypeError for 
> a class whose __init__
>    takes arguments, for df.schema, df.dtypes and df.collect().
> 3. A cast to a UDT of its own simpleString() shows UDT. PySpark's 
> CastExpression.__repr__ shows
>    simpleString().upper() of the type object 
> (python/pyspark/sql/connect/expressions.py:1004-1016).
>    The same goes for a type of a Python subclass, and for the 
> pyspark.mllib.linalg VectorUDT and
>    MatrixUDT, of which the core names only the pyspark.ml.linalg classes.
> 4. DataFrame.to of a struct caches the core's type rather than the struct as 
> given, as PySpark's
>    res._cached_schema = schema does.
> Repro (master 50594b2e, against a Spark 4.3.0 server)
>   >>> ArrayType(UserDefinedType())
>   PySparkNotImplementedError: [NOT_IMPLEMENTED] module() is not implemented.
>   >>> u = PtUDT(); u.marker = 1
>   >>> df = spark.createDataFrame([(Pt(1.0, 2.0),)], 
> StructType([StructField("p", u)]))
>   >>> df.schema["p"].dataType is u
>   True
>   >>> spark.createDataFrame([(Pt(1.0, 2.0),)], StructType([StructField("p", 
> ArgUDT(2))])).schema
>   StructType([StructField('p', ArgUDT(), True)])        # ArgUDT.__init__ 
> takes a scale
>   >>> F.col("x").cast(ScaleUDT(2))
>   Column<'CAST(x AS UDT)'>
>   >>> F.col("x").cast(pyspark.mllib.linalg.VectorUDT())
>   Column<'CAST(x AS UDT)'>
> Expected (as with pyspark-client 4.3.0)
>   ArrayType(UserDefinedType(), True), and its json() raises the 
> NOT_IMPLEMENTED of module()
>   False (a new PtUDT(), without the attribute)
>   TypeError: ArgUDT.__init__() missing 1 required positional argument: 'scale'
>   Column<'CAST(x AS SCALE2)'>
>   Column<'CAST(x AS VECTOR)'>



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to