[
https://issues.apache.org/jira/browse/SPARK-61758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61758:
------------------------------------
Assignee: Hyukjin Kwon
> Fix the PySpark differences that the review of the last 200 commits found in
> the types, Columns, windows, table arguments, UDF arguments and copies of the
> Python client, and the gaps of its CI
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61758
> URL: https://issues.apache.org/jira/browse/SPARK-61758
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The review of the last 200 commits of spark-connect-rust found these
> differences from PySpark 4.3.0 in the Python client (outside ML), and gaps of
> its CI. This fixes them:
> - JSON type strings are parsed as PySpark's prefix regexes parse them; null
> and calendarinterval are rejected as PySpark rejects them;
> CalendarIntervalType pickles as "interval"; the hash of a DataType is
> hash(str(self)), as PySpark's.
> - The schema of a server UDT keeps its sqlType (df.schema.json(),
> createDataFrame([], df.schema)).
> - No origin on the CASE WHEN `when`; the origins of between, getItem, asc and
> desc are those of PySpark's dunders; alias and name take bytes and give
> protobuf's TypeError.
> - TableArg checks its state before its columns, and its copies are PySpark's;
> the literal arguments of a UDF raise PySpark's AssertionError; the UDTF
> `name` that PySpark does not have is removed.
> - The copies of Column, GroupedData, Catalog and Window keep their `__dict__`.
> - The runtime abort is documented and tested; the core integration test runs
> in the connect-e2e job.
> - CI guards that every e2e file runs, and the path filters of the docs and
> wheel jobs.
> - Docs: deep plans, call-time conversions, the pickle of a TableArg, the
> order of getMessageParameters, char/varchar.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]