[
https://issues.apache.org/jira/browse/SPARK-60906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60906:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Make str(), repr() and fromDDL of Spark Connect Rust data types match PySpark
> -----------------------------------------------------------------------------
>
> Key: SPARK-60906
> URL: https://issues.apache.org/jira/browse/SPARK-60906
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Data types of the Rust-backed Python client differ from PySpark Connect 4.3.0
> in three related ways:
> * PySpark data types define only {{__repr__}}, so {{str()}} of a data type is
> its repr: {{str(LongType())}} is {{LongType()}}. The client's base
> {{DataType}} defines {{__str__}} as the simple string, {{bigint}}, so
> {{str()}} of every data type differs, as do error messages that format a type
> with {{str()}}.
> * The base {{DataType.__repr__}} returns the simple string of its unset type,
> so {{repr()}} of a bare {{DataType()}}, or of a {{UserDefinedType}} without
> its own {{__repr__}}, is {{void}}. PySpark gives the class name, such as
> {{DataType()}} and {{ExamplePointUDT()}}.
> * {{DataType.fromDDL}} returns a bare {{DataType}} rather than the concrete
> type. PySpark returns, for example, a {{StructType}} for {{a int, b string}}
> and a {{DecimalType}} for {{decimal(10,10)}}.
> {{repr()}} of the concrete data types already matches PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
