[ 
https://issues.apache.org/jira/browse/SPARK-61440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61440:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the retries of the calls of the Spark Connect Rust client in one place, 
> and do not panic on a stream deadline too far to be a time
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61440
>                 URL: https://issues.apache.org/jira/browse/SPARK-61440
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> The Rust client (crates/spark-connect-core/src/client.rs) retries its calls 
> in four hand-written loops: the unary and stream-opening calls (with_retry), 
> the artifact upload, the background ReleaseExecute, and the reattach of a 
> result stream. Each has its own copy of the decision "can the policy retry 
> this failure, is a retry left, wait its backoff", so a change to the retry 
> policy has to be made four times, and the loops can drift apart.
> This makes that decision in one place in retries.rs, as PySpark's Retrying: a 
> retry of the policy with its backoff, and the retry made at once that PySpark 
> raises as a RetryException, bounded by its elapsed time. The differences 
> between the loops that are on purpose stay, as parameters: the calls of a 
> client check that the client is still open before a retry and ReleaseExecute 
> does not, an attempt of an upload can fail with an error that is not retried, 
> and the reader of a reattachable stream shares the attempts of the policy 
> among the failures of a response (SPARK-61415).
> Two related fixes in the same code:
> * The deadline of a stream is computed as Instant::now() + deadline at three 
> places, which panics for a configured deadline too large to add to the time. 
> A deadline that cannot be a time is never reached, so it is none.
> * INVALID_CURSOR.DISCONNECTED and INVALID_HANDLE.OPERATION_NOT_FOUND / 
> SESSION_NOT_FOUND are recognized by their name in the message of the error, 
> as PySpark does. The match stays, in one named function with tests.
> No other change of behavior.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to