[
https://issues.apache.org/jira/browse/SPARK-60918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60918:
------------------------------------
Assignee: L. C. Hsieh
> Make == and hash() of Spark Connect Rust data types match PySpark
> -----------------------------------------------------------------
>
> Key: SPARK-60918
> URL: https://issues.apache.org/jira/browse/SPARK-60918
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In the Rust-backed Python drop-in, the data type classes ({{DataType}} and
> every concrete type such as {{LongType}}, {{StructType}}) define no
> {{__eq__}}/{{__hash__}}, so equality falls back to object identity:
> {code:python}
> LongType() == LongType() # False (PySpark: True)
> IntegerType() == DataType.fromDDL("int") # False (PySpark: True)
> {code}
> {{StructField}} defines {{__eq__}} but no {{__hash__}}, so it is unhashable,
> while PySpark's is hashable.
> PySpark defines {{DataType.__eq__}} as {{isinstance(other, self.__class__)
> and self.__dict__ == other.__dict__}} and {{__hash__}} as
> {{hash(str(self))}}. The drop-in should match: compare by the type built from
> the live fields (a StructType mutated with {{add()}} compares by its current
> fields), keep PySpark's rule for pure-Python {{UserDefinedType}} subclasses,
> and keep {{hash}} consistent with {{==}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
