[
https://issues.apache.org/jira/browse/SPARK-60907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60907:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send reader schemas in Spark Connect Rust as PySpark does, and accept a
> StructType in schema()
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-60907
> URL: https://issues.apache.org/jira/browse/SPARK-60907
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{DataFrameReader.schema}} and
> {{DataStreamReader.schema}} take a StructType, which is sent as its JSON, or
> a string, which is sent as is; anything else raises {{NOT_EXPECTED_TYPE}}.
> The {{schema}} arguments of {{load()}}, {{csv()}}, {{json()}} and the other
> format methods go through the same method.
> The Rust-backed Python client differs:
> * {{schema()}} accepts only a string, so {{spark.read.schema(struct_type)}}
> and {{spark.readStream.schema(struct_type)}} raise {{TypeError}}.
> * The {{schema}} arguments of the format methods are parsed on the client and
> sent as a simple string. A StructType with field metadata, such as a comment,
> then fails on the server with {{PARSE_SYNTAX_ERROR}}, and a DDL string that
> the client parser does not accept, such as one with {{COMMENT}}, fails with
> {{ValueError}}. Data types other than a StructType are accepted rather than
> raising {{NOT_EXPECTED_TYPE}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
