Hyukjin Kwon created SPARK-60912:
------------------------------------
Summary: Make Row in Spark Connect Rust behave as PySpark's Row
Key: SPARK-60912
URL: https://issues.apache.org/jira/browse/SPARK-60912
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's {{Row}} is a {{tuple}} subclass. The Rust-backed Python client's
{{Row}} differs from it in these PySpark 4.3.0 results:
* {{repr()}} and {{str()}} give {{Row(x=1, y='a')}}, with the {{repr()}} of
each value, and {{<Row(1, 2)>}} for a row without field names. The client gives
{{[1, a]}} and {{[1, 2]}}, and shows a date as its day number, {{None}} as
{{NULL}} and bytes as a list of numbers. So {{print(spark.range(1).collect())}}
prints {{[[0]]}}, not {{[Row(id=0)]}}.
* {{Row("a", "b")}} is a row class: {{Row("a", "b")(1, 2)}} gives {{Row(a=1,
b=2)}}. The client raises {{TypeError}}, as its row is not callable.
* A row is hashable, orders as the tuple of its values does, as in
{{sorted(rows)}}, equals that tuple, and pickles. The client raises
{{TypeError}} for {{hash()}}, {{<}} and {{pickle.dumps()}}, and {{Row(x=1) ==
(1,)}} is false.
* Slicing and concatenation give tuples, as in {{r[0:1]}} and {{r + (3,)}}. The
client raises {{KeyError}} and {{TypeError}}.
* {{"x" in r}} tests the field names of a row that has them. The client tests
the values.
* {{Row(1, 2).asDict()}} raises {{CANNOT_CONVERT_TYPE}}. The client returns a
dict keyed by {{_1}} and {{_2}}.
* A missing field raises {{PySparkAttributeError}}
({{ATTRIBUTE_NOT_SUPPORTED}}) for {{r.missing}}, and {{PySparkValueError}} for
{{r["missing"]}}. The client raises a plain {{AttributeError}} and a
{{KeyError}}.
Make the client's {{Row}} behave as PySpark's. {{isinstance(r, tuple)}} stays
false, as a PyO3 class cannot subclass {{tuple}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
