[
https://issues.apache.org/jira/browse/SPARK-61367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61367:
------------------------------------
Assignee: Hyukjin Kwon
> Match PySpark and the server in the Rust API's pivot literals, year-month
> intervals, range repartitions, sample_by strata, DDL, Arrow batch checks,
> release_session, listener panics and handler threads
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61367
> URL: https://issues.apache.org/jira/browse/SPARK-61367
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Low-priority findings of the s16 review of the Rust API
> (apache-spark-connect), each checked against official PySpark 4.3.0, the
> server or the Scala client:
> - pivot sent a Float value as a double literal, so its column was named
> "1.100000023841858" where the pivot without values (and the Scala client)
> names it "1.1"; Byte and Short were sent as ints; and its NOT_EXPECTED_TYPE
> message named only bool, float, int and str.
> - Value had no year-month interval: collect gave Value::String("1-0"), so a
> pivot on collected values named the column "1-0", not "INTERVAL '1-0' YEAR TO
> MONTH".
> - repartition_by_range with no columns was split round-robin by the server;
> PySpark raises CANNOT_BE_EMPTY (and VALUE_NOT_POSITIVE for a count that is
> not positive). sample_by turned a stratum that is not a literal into an unset
> one; PySpark raises DISALLOWED_TYPE_FOR_CONTAINER.
> - DataType::from_ddl rejected whitespace before "<" (array <int>, STRUCT
> <a:int>), which the server accepts.
> - PySpark checks the start_offset and the row count of each Arrow batch of a
> result; ours did not, and grew the joined buffer of chunked batches without
> reserving it.
> - SparkSession::release_session after stop() or close() still sent
> ReleaseSession; PySpark raises NO_ACTIVE_SESSION.
> - A native streaming listener's panic on an asynchronous event failed the
> whole subscription and cleared every listener; PySpark's post_to_all isolates
> each listener.
> - In a current_thread Tokio runtime, the progress handlers' update calls ran
> on a helper thread; PySpark calls them on the thread that iterates the
> responses.
> - With several invalid UTF-8 values in one struct or map column, ours
> reported the first row by row; PySpark reports field by field, keys before
> values.
> - A Raw relation (a plan of pyspark.ml) counted as one level for the depth
> bound whatever its nesting, and cloning one at the bound overflowed a small
> stack; LogicalPlan::to_proto has no bounded variant.
> - Docs: the E0509 of moving fields out of a LogicalPlan (it implements Drop),
> and that is_local() is a client-side guess.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]