[ 
https://issues.apache.org/jira/browse/SPARK-61367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61367:
------------------------------------

    Assignee: Hyukjin Kwon

> Match PySpark and the server in the Rust API's pivot literals, year-month 
> intervals, range repartitions, sample_by strata, DDL, Arrow batch checks, 
> release_session, listener panics and handler threads
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61367
>                 URL: https://issues.apache.org/jira/browse/SPARK-61367
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Low-priority findings of the s16 review of the Rust API 
> (apache-spark-connect), each checked against official PySpark 4.3.0, the 
> server or the Scala client:
> - pivot sent a Float value as a double literal, so its column was named 
> "1.100000023841858" where the pivot without values (and the Scala client) 
> names it "1.1"; Byte and Short were sent as ints; and its NOT_EXPECTED_TYPE 
> message named only bool, float, int and str.
> - Value had no year-month interval: collect gave Value::String("1-0"), so a 
> pivot on collected values named the column "1-0", not "INTERVAL '1-0' YEAR TO 
> MONTH".
> - repartition_by_range with no columns was split round-robin by the server; 
> PySpark raises CANNOT_BE_EMPTY (and VALUE_NOT_POSITIVE for a count that is 
> not positive). sample_by turned a stratum that is not a literal into an unset 
> one; PySpark raises DISALLOWED_TYPE_FOR_CONTAINER.
> - DataType::from_ddl rejected whitespace before "<" (array <int>, STRUCT 
> <a:int>), which the server accepts.
> - PySpark checks the start_offset and the row count of each Arrow batch of a 
> result; ours did not, and grew the joined buffer of chunked batches without 
> reserving it.
> - SparkSession::release_session after stop() or close() still sent 
> ReleaseSession; PySpark raises NO_ACTIVE_SESSION.
> - A native streaming listener's panic on an asynchronous event failed the 
> whole subscription and cleared every listener; PySpark's post_to_all isolates 
> each listener.
> - In a current_thread Tokio runtime, the progress handlers' update calls ran 
> on a helper thread; PySpark calls them on the thread that iterates the 
> responses.
> - With several invalid UTF-8 values in one struct or map column, ours 
> reported the first row by row; PySpark reports field by field, keys before 
> values.
> - A Raw relation (a plan of pyspark.ml) counted as one level for the depth 
> bound whatever its nesting, and cloning one at the bound overflowed a small 
> stack; LogicalPlan::to_proto has no bounded variant.
> - Docs: the E0509 of moving fields out of a LogicalPlan (it implements Drop), 
> and that is_local() is a client-side guess.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to