Hyukjin Kwon created SPARK-61415:
------------------------------------
Summary: Back off and bound the reattaches of a result stream that
keeps failing, and raise NO_ACTIVE_SESSION when the plan compression lookup
closed the client, in Spark Connect Rust as PySpark does
Key: SPARK-61415
URL: https://issues.apache.org/jira/browse/SPARK-61415
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Three differences of Spark Connect Rust's transport from PySpark 4.3.0:
1. ReattachableResponseStream::message has no backoff and no retry limit for a
stream that fails after it opens. When a reattach gets its response headers and
the stream then fails before the next response, the client reattaches again at
once, without waiting or counting. PySpark's
ExecutePlanResponseReattachableIterator._has_next runs the reattach and the
read of its stream in one attempt of its Retrying, so each such failure waits
the policy's backoff and is raised after max_retries. Through a proxy that
resets every connection after 100 KB, a collect of a 300 KB row reattached 197
times in 1.3 s; PySpark reattached 4 times over 4.7 s.
2. When the plan compression lookup of the first plan of a client closes the
client (the server restarted, so the server-side session ID changed), the
client still sends the ExecutePlan or AnalyzePlan. PySpark raises
NO_ACTIVE_SESSION and sends nothing.
3. The FetchErrorDetails and stack-trace Config requests sent while raising a
server error carry no user context extensions (PySpark's carry them), and the
AddArtifacts and ArtifactStatus requests carry client_type, which PySpark's
ArtifactManager does not send. docs/compatibility.md also gives the wrong old
codes of GOAWAY failures.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]