Hyukjin Kwon created SPARK-61916:
------------------------------------

             Summary: Raise PySpark's errors for abstract data types given to 
UDTF return types, registerJavaFunction, table schemas and createDataFrame
                 Key: SPARK-61916
                 URL: https://issues.apache.org/jira/browse/SPARK-61916
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


After SPARK-61859, the base data type classes such as AtomicType() can be 
instantiated, as in PySpark, and PySpark's conversion of a type to its proto 
raises UNSUPPORTED_OPERATION for them, also when nested. Four places of the 
drop-in that take a type do not use that conversion, so a nested abstract type 
reaches the server as a DataType.Unparsed of its JSON, where PySpark raises in 
the client:

- the return type of udtf(...)() and of spark.udtf.register;
- the return type of spark.udf.registerJavaFunction;
- the schema of spark.catalog.createTable.

For example, udtf(Fn, returnType=StructType([StructField("a", 
AtomicType())]))().schema raises the server's 
CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST {"kindCase": 
"UNPARSED"} on the drop-in, and PySpark's UNSUPPORTED_OPERATION {"operation": 
"data type AtomicType()"} on pyspark-client 4.3.0. registerJavaFunction("f", 
"cls", AtomicType()) raises PARSE_SYNTAX_ERROR. The same already happened for a 
StructField nested as a type.

Also, createDataFrame with an abstract type as its schema classifies the schema 
by the core type rather than the Python object: createDataFrame([1], 
AtomicType()) raises NOT_EXPECTED_TYPE {"arg_type": "UnparsedDataType"}, where 
PySpark raises UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION {"data_type": 
"AtomicType()"}, and createDataFrame([1], DataType()) names "UnparsedDataType" 
where PySpark names "DataType".

These should raise PySpark's errors, as cast and try_cast already do.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to