[
https://issues.apache.org/jira/browse/SPARK-61415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61415:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Back off and bound the reattaches of a result stream that keeps failing, and
> raise NO_ACTIVE_SESSION when the plan compression lookup closed the client,
> in Spark Connect Rust as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61415
> URL: https://issues.apache.org/jira/browse/SPARK-61415
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Three differences of Spark Connect Rust's transport from PySpark 4.3.0:
> 1. ReattachableResponseStream::message has no backoff and no retry limit for
> a stream that fails after it opens. When a reattach gets its response headers
> and the stream then fails before the next response, the client reattaches
> again at once, without waiting or counting. PySpark's
> ExecutePlanResponseReattachableIterator._has_next runs the reattach and the
> read of its stream in one attempt of its Retrying, so each such failure waits
> the policy's backoff and is raised after max_retries. Through a proxy that
> resets every connection after 100 KB, a collect of a 300 KB row reattached
> 197 times in 1.3 s; PySpark reattached 4 times over 4.7 s.
> 2. When the plan compression lookup of the first plan of a client closes the
> client (the server restarted, so the server-side session ID changed), the
> client still sends the ExecutePlan or AnalyzePlan. PySpark raises
> NO_ACTIVE_SESSION and sends nothing.
> 3. The FetchErrorDetails and stack-trace Config requests sent while raising a
> server error carry no user context extensions (PySpark's carry them), and the
> AddArtifacts and ArtifactStatus requests carry client_type, which PySpark's
> ArtifactManager does not send. docs/compatibility.md also gives the wrong old
> codes of GOAWAY failures.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]