[
https://issues.apache.org/jira/browse/SPARK-61094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61094:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Add UnparsedDataType of pyspark.sql.connect.types, the returnType of a UDTF
> of a DDL string, in the Python client as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61094
> URL: https://issues.apache.org/jira/browse/SPARK-61094
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{pyspark.sql.connect.types.UnparsedDataType}} is the
> type of a DDL string that the server parses: its {{simpleString}} is
> "unparsed('<ddl>')", its repr "UnparsedDataType('<ddl>')", and its
> {{jsonValue}}, {{needConversion}}, {{toInternal}} and {{fromInternal}} raise
> {{INVALID_CALL_ON_UNRESOLVED_OBJECT}}. The {{returnType}} of a UDTF of a DDL
> string is an {{UnparsedDataType}}, and {{spark.dataSource.register}} of a
> class without a {{name}} raises the {{AttributeError}} of {{name()}}.
> The Rust-backed Python client lacks {{pyspark.sql.connect.types}}, keeps the
> DDL string as the {{returnType}} of a UDTF, and raises a {{TypeError}}
> ("DataSource class must have a classmethod name()") of {{register}}. Add them
> as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
