[ 
https://issues.apache.org/jira/browse/SPARK-61911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61911:
------------------------------------

    Assignee: Hyukjin Kwon

> Raise the remaining argument errors where PySpark raises them, and send MERGE 
> assignments in their order
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61911
>                 URL: https://issues.apache.org/jira/browse/SPARK-61911
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61880 from a round-2 review against PySpark 4.3.0:
> - zipWithIndex converts its column name at the call, where PySpark sets it at 
> plan build (bytes work there), and zipWithIndex and toJSON send no col and 
> alias origins.
> - applyInPandasWithState, transformWithStateInPandas and transformWithState 
> convert their modes at the call; transformWithState sends its output type 
> parsed, and both TWS methods drop event_time_column_name "".
> - The avro and protobuf functions take their schema, message name and 
> descriptor as Rust types, where PySpark wraps them in lit().
> - mapInPandas and mapInArrow take only a bool barrier at the call, and send 
> the resource profile's create command at the call, where PySpark sets the 
> flag and reads the profile at plan build.
> - nearestByJoin, readStream.name, Column and SparkSession __getattr__, the 
> connect Builder.master, registerJavaFunction, udtf.register, 
> udf.register(None, ...), interruptTag and cogroup(5) raise other errors than 
> PySpark's, or refuse what PySpark takes.
> - MergeIntoWriter checks its arguments at the call, and sends its 
> assignments, and raises their errors, in a random HashMap order, where 
> PySpark keeps the dict's order.
> - An UNSUPPORTED_OPERATION message loses a lone surrogate; spark.client's 
> copies and pickles fail with other errors than PySpark's; StorageLevel does 
> not pickle at protocols 0 and 1.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to