[
https://issues.apache.org/jira/browse/SPARK-60739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60739:
------------------------------------
Assignee: Hyukjin Kwon
> Fix truncated results and unreleased executions in the Rust client's
> reattachable execution, and four drop-in parity gaps
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60739
> URL: https://issues.apache.org/jira/browse/SPARK-60739
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust client's reattachable execution and four drop-in APIs do not match
> the reference PySpark client (Spark 4.3.0). The official parity tests run the
> reference pyspark over the Rust transport, so they do not exercise these.
> *Reattachable execution* (spark-connect-core, used by every DataFrame action
> of the native API and the drop-in):
> - The server completes the response stream of a reattachable execution before
> the result is complete, on purpose, after
> spark.connect.execute.reattachable.senderMaxStreamDuration (2m) or
> senderMaxStreamSize (1g); the client must reattach to read the rest. The Rust
> client takes the end of the stream for the end of the result, so it silently
> returns a truncated result: with a 1 MiB senderMaxStreamSize, collect() of
> range(300000) returns 135000 rows, and a 3-second query on a server with a 1s
> senderMaxStreamDuration returns no rows and count() = 0.
> - It never sends ReleaseExecute. The server keeps every finished execution,
> with its request and up to 10 MiB of its responses, until
> spark.connect.execute.manager.detachedTimeout (5m) or until the session
> closes. With large local relations this exhausted a 1 GB server heap:
> spark.stop() failed with "Java heap space".
> *Drop-in API parity*:
> - StructType.add() does not update the DataType that mapInArrow/mapInPandas
> (and the grouped applyIn* functions) resolve their schema from, so
> df.mapInArrow(f, StructType().add("a", "long").add("b", "long")) has no
> columns; str() of such a StructType is "struct<>".
> - DataFrame.repartitionById(n, "col") takes the column name for a string
> literal, and fails analysis.
> - RuntimeConf.get(key, None) raises SQL_CONF_NOT_FOUND for an unset key
> instead of returning None.
> - SparkSession.createDataFrame(pyarrow.Table) raises TypeError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
