Hyukjin Kwon created SPARK-60906:
------------------------------------
Summary: Make str(), repr() and fromDDL of Spark Connect Rust data
types match PySpark
Key: SPARK-60906
URL: https://issues.apache.org/jira/browse/SPARK-60906
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Data types of the Rust-backed Python client differ from PySpark Connect 4.3.0
in three related ways:
* PySpark data types define only {{__repr__}}, so {{str()}} of a data type is
its repr: {{str(LongType())}} is {{LongType()}}. The client's base {{DataType}}
defines {{__str__}} as the simple string, {{bigint}}, so {{str()}} of every
data type differs, as do error messages that format a type with {{str()}}.
* The base {{DataType.__repr__}} returns the simple string of its unset type,
so {{repr()}} of a bare {{DataType()}}, or of a {{UserDefinedType}} without its
own {{__repr__}}, is {{void}}. PySpark gives the class name, such as
{{DataType()}} and {{ExamplePointUDT()}}.
* {{DataType.fromDDL}} returns a bare {{DataType}} rather than the concrete
type. PySpark returns, for example, a {{StructType}} for {{a int, b string}}
and a {{DecimalType}} for {{decimal(10,10)}}.
{{repr()}} of the concrete data types already matches PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
