L. C. Hsieh created SPARK-60918:
-----------------------------------
Summary: Make == and hash() of Spark Connect Rust data types match
PySpark
Key: SPARK-60918
URL: https://issues.apache.org/jira/browse/SPARK-60918
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
Fix For: connect-rust-4.3.0
In the Rust-backed Python drop-in, the data type classes ({{DataType}} and
every concrete type such as {{LongType}}, {{StructType}}) define no
{{__eq__}}/{{__hash__}}, so equality falls back to object identity:
{code:python}
LongType() == LongType() # False (PySpark: True)
IntegerType() == DataType.fromDDL("int") # False (PySpark: True)
{code}
{{StructField}} defines {{__eq__}} but no {{__hash__}}, so it is unhashable,
while PySpark's is hashable.
PySpark defines {{DataType.__eq__}} as {{isinstance(other, self.__class__) and
self.__dict__ == other.__dict__}} and {{__hash__}} as {{hash(str(self))}}. The
drop-in should match: compare by the type built from the live fields (a
StructType mutated with {{add()}} compares by its current fields), keep
PySpark's rule for pure-Python {{UserDefinedType}} subclasses, and keep
{{hash}} consistent with {{==}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
