Hyukjin Kwon created SPARK-61415:
------------------------------------

             Summary: Back off and bound the reattaches of a result stream that 
keeps failing, and raise NO_ACTIVE_SESSION when the plan compression lookup 
closed the client, in Spark Connect Rust as PySpark does
                 Key: SPARK-61415
                 URL: https://issues.apache.org/jira/browse/SPARK-61415
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Three differences of Spark Connect Rust's transport from PySpark 4.3.0:

1. ReattachableResponseStream::message has no backoff and no retry limit for a 
stream that fails after it opens. When a reattach gets its response headers and 
the stream then fails before the next response, the client reattaches again at 
once, without waiting or counting. PySpark's 
ExecutePlanResponseReattachableIterator._has_next runs the reattach and the 
read of its stream in one attempt of its Retrying, so each such failure waits 
the policy's backoff and is raised after max_retries. Through a proxy that 
resets every connection after 100 KB, a collect of a 300 KB row reattached 197 
times in 1.3 s; PySpark reattached 4 times over 4.7 s.

2. When the plan compression lookup of the first plan of a client closes the 
client (the server restarted, so the server-side session ID changed), the 
client still sends the ExecutePlan or AnalyzePlan. PySpark raises 
NO_ACTIVE_SESSION and sends nothing.

3. The FetchErrorDetails and stack-trace Config requests sent while raising a 
server error carry no user context extensions (PySpark's carry them), and the 
AddArtifacts and ArtifactStatus requests carry client_type, which PySpark's 
ArtifactManager does not send. docs/compatibility.md also gives the wrong old 
codes of GOAWAY failures.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to