Hyukjin Kwon created SPARK-61758:
------------------------------------

             Summary: Fix the PySpark differences that the review of the last 
200 commits found in the types, Columns, windows, table arguments, UDF 
arguments and copies of the Python client, and the gaps of its CI
                 Key: SPARK-61758
                 URL: https://issues.apache.org/jira/browse/SPARK-61758
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The review of the last 200 commits of spark-connect-rust found these 
differences from PySpark 4.3.0 in the Python client (outside ML), and gaps of 
its CI. This fixes them:

- JSON type strings are parsed as PySpark's prefix regexes parse them; null and 
calendarinterval are rejected as PySpark rejects them; CalendarIntervalType 
pickles as "interval"; the hash of a DataType is hash(str(self)), as PySpark's.
- The schema of a server UDT keeps its sqlType (df.schema.json(), 
createDataFrame([], df.schema)).
- No origin on the CASE WHEN `when`; the origins of between, getItem, asc and 
desc are those of PySpark's dunders; alias and name take bytes and give 
protobuf's TypeError.
- TableArg checks its state before its columns, and its copies are PySpark's; 
the literal arguments of a UDF raise PySpark's AssertionError; the UDTF `name` 
that PySpark does not have is removed.
- The copies of Column, GroupedData, Catalog and Window keep their `__dict__`.
- The runtime abort is documented and tested; the core integration test runs in 
the connect-e2e job.
- CI guards that every e2e file runs, and the path filters of the docs and 
wheel jobs.
- Docs: deep plans, call-time conversions, the pickle of a TableArg, the order 
of getMessageParameters, char/varchar.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to