Hyukjin Kwon created SPARK-61758:
------------------------------------
Summary: Fix the PySpark differences that the review of the last
200 commits found in the types, Columns, windows, table arguments, UDF
arguments and copies of the Python client, and the gaps of its CI
Key: SPARK-61758
URL: https://issues.apache.org/jira/browse/SPARK-61758
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The review of the last 200 commits of spark-connect-rust found these
differences from PySpark 4.3.0 in the Python client (outside ML), and gaps of
its CI. This fixes them:
- JSON type strings are parsed as PySpark's prefix regexes parse them; null and
calendarinterval are rejected as PySpark rejects them; CalendarIntervalType
pickles as "interval"; the hash of a DataType is hash(str(self)), as PySpark's.
- The schema of a server UDT keeps its sqlType (df.schema.json(),
createDataFrame([], df.schema)).
- No origin on the CASE WHEN `when`; the origins of between, getItem, asc and
desc are those of PySpark's dunders; alias and name take bytes and give
protobuf's TypeError.
- TableArg checks its state before its columns, and its copies are PySpark's;
the literal arguments of a UDF raise PySpark's AssertionError; the UDTF `name`
that PySpark does not have is removed.
- The copies of Column, GroupedData, Catalog and Window keep their `__dict__`.
- The runtime abort is documented and tested; the core integration test runs in
the connect-e2e job.
- CI guards that every e2e file runs, and the path filters of the docs and
wheel jobs.
- Docs: deep plans, call-time conversions, the pickle of a TableArg, the order
of getMessageParameters, char/varchar.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]