[
https://issues.apache.org/jira/browse/SPARK-60997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60997:
------------------------------------
Assignee: Hyukjin Kwon
> Apply the per-RPC deadlines of PySpark Connect in the Rust client, and
> reattach an execution past its deadline
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60997
> URL: https://issues.apache.org/jira/browse/SPARK-60997
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect 4.3.0 applies per-RPC deadlines, {{RpcDeadlines}} of
> {{pyspark.sql.connect.client}}: 10 minutes for each stream of a reattachable
> {{ExecutePlan}} and each {{ReattachExecute}}, an hour for {{AnalyzePlan}} and
> {{AddArtifacts}}, and 10 minutes for {{Config}}, {{Interrupt}},
> {{ReleaseSession}}, {{ArtifactStatus}}, {{CloneSession}}, {{GetStatus}} and
> {{FetchErrorDetails}}. When the deadline of an execution's stream fires, the
> operation continues on the server and the client opens a new
> {{ReattachExecute}} stream to resume its results
> ({{ExecutePlanResponseReattachableIterator}}); a {{DEADLINE_EXCEEDED}} of
> another call raises {{SparkConnectGrpcException}}. So a call on a connection
> that broke without an error does not wait forever.
> The Rust client sets no deadline on its calls, so such a call waits until the
> connection fails, and a {{DEADLINE_EXCEEDED}} of an execution's stream, as
> from a proxy, fails the execution where PySpark reattaches. Apply the
> deadlines of PySpark 4.3.0 and reattach an execution whose stream exceeds its
> deadline.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
