[ 
https://issues.apache.org/jira/browse/SPARK-61411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Meehan updated SPARK-61411:
-------------------------------
    Fix Version/s: connect-rust-4.3.0

> Retry the errors that carry a RetryInfo, after the delay the server asks for, 
> as PySpark does
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61411
>                 URL: https://issues.apache.org/jira/browse/SPARK-61411
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Tim Meehan
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> The Rust client does not retry an error whose status details carry a 
> {{google.rpc.RetryInfo}}, and no retry of it waits for a delay that the 
> server asks for. PySpark's {{DefaultPolicy}} retries any error that carries a 
> {{RetryInfo}}, whatever its status code, and waits at least the 
> {{retry_delay}} of it (at most 10 minutes) before the next attempt.
> A server that throttles with {{RESOURCE_EXHAUSTED}} and a {{RetryInfo}} of 
> 400 ms is the case. Consider a loopback gRPC service that answers Config with 
> that status once and then succeeds. The PySpark behavior below is read from 
> its source (pyspark-client 4.3.0), not from a run; the Rust behavior is from 
> a test of the service at spark-connect-rust master 125654b:
> * pyspark-client 4.3.0 (read from {{pyspark/sql/connect/client/retries.py}}, 
> {{DefaultPolicy.can_retry}} and {{RetryPolicyState.next_attempt}}): retries, 
> and waits at least 400 ms.
> * spark-connect-rust master 125654b, with a Rust test of the same service 
> ({{crates/spark-connect-core/tests/retry_info.rs}}, added by the fix): the 
> error is raised at the first call; the server sees one call.
> {code:rust}
> // The service fails Config once with RESOURCE_EXHAUSTED and a RetryInfo of 
> 400 ms.
> let result = client.config(request).await; // Err at 125654b; Ok after >= 400 
> ms with the fix
> {code}
> The same holds for every call that goes through the retry policy (unary 
> calls, AddArtifacts, ReleaseExecute, and the reattach of an execution). I did 
> not run a stock OSS Spark 4.3.0 Connect server for it: the server does not 
> send a {{RetryInfo}} on demand.
> From source, at 125654b:
> * {{RetryPolicy::can_retry}} (crates/spark-connect-core/src/retries.rs:76) 
> retries only on the status code (UNAVAILABLE, a cursor disconnect, and two 
> transport errors); it does not look at the details.
> * Every caller passes no server delay: {{state.next_attempt(None)}} at 
> crates/spark-connect-core/src/client.rs:1151, 1908, 2101 and 2283. 
> {{RetryPolicyState::next_attempt}} (retries.rs:135) takes the delay and 
> clamps it to {{max_server_retry_delay_ms}}, but nothing supplies it.
> * {{RetryPolicy::default()}} has {{recognize_server_retry_delay: false}} and 
> no {{max_server_retry_delay_ms}} (retries.rs:45), where PySpark's 
> {{DefaultPolicy}} has {{recognize_server_retry_delay=True}} and 
> {{max_server_retry_delay}} of 10 minutes.
> * A stream that fails after its headers with a retryable error is reattached 
> at once, with no wait and no retry counted, where PySpark raises the error 
> inside its {{Retrying}}, which waits and counts it (client.rs:2229-2236 at 
> 125654b).
> Known difference, not part of this issue: the other defaults of 
> {{RetryPolicy::default()}} (initial backoff, multiplier, maximum backoff, 
> jitter) differ from PySpark's {{DefaultPolicy}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to