Tim Meehan created SPARK-61411:
----------------------------------
Summary: Retry the errors that carry a RetryInfo, after the delay
the server asks for, as PySpark does
Key: SPARK-61411
URL: https://issues.apache.org/jira/browse/SPARK-61411
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
The Rust client does not retry an error whose status details carry a
{{google.rpc.RetryInfo}}, and no retry of it waits for a delay that the server
asks for. PySpark's {{DefaultPolicy}} retries any error that carries a
{{RetryInfo}}, whatever its status code, and waits at least the {{retry_delay}}
of it (at most 10 minutes) before the next attempt.
A server that throttles with {{RESOURCE_EXHAUSTED}} and a {{RetryInfo}} of 400
ms is the case. Consider a loopback gRPC service that answers Config with that
status once and then succeeds. The PySpark behavior below is read from its
source (pyspark-client 4.3.0), not from a run; the Rust behavior is from a test
of the service at spark-connect-rust master 125654b:
* pyspark-client 4.3.0 (read from {{pyspark/sql/connect/client/retries.py}},
{{DefaultPolicy.can_retry}} and {{RetryPolicyState.next_attempt}}): retries,
and waits at least 400 ms.
* spark-connect-rust master 125654b, with a Rust test of the same service
({{crates/spark-connect-core/tests/retry_info.rs}}, added by the fix): the
error is raised at the first call; the server sees one call.
{code:rust}
// The service fails Config once with RESOURCE_EXHAUSTED and a RetryInfo of 400
ms.
let result = client.config(request).await; // Err at 125654b; Ok after >= 400
ms with the fix
{code}
The same holds for every call that goes through the retry policy (unary calls,
AddArtifacts, ReleaseExecute, and the reattach of an execution). I did not run
a stock OSS Spark 4.3.0 Connect server for it: the server does not send a
{{RetryInfo}} on demand.
>From source, at 125654b:
* {{RetryPolicy::can_retry}} (crates/spark-connect-core/src/retries.rs:76)
retries only on the status code (UNAVAILABLE, a cursor disconnect, and two
transport errors); it does not look at the details.
* Every caller passes no server delay: {{state.next_attempt(None)}} at
crates/spark-connect-core/src/client.rs:1151, 1908, 2101 and 2283.
{{RetryPolicyState::next_attempt}} (retries.rs:135) takes the delay and clamps
it to {{max_server_retry_delay_ms}}, but nothing supplies it.
* {{RetryPolicy::default()}} has {{recognize_server_retry_delay: false}} and no
{{max_server_retry_delay_ms}} (retries.rs:45), where PySpark's
{{DefaultPolicy}} has {{recognize_server_retry_delay=True}} and
{{max_server_retry_delay}} of 10 minutes.
* A stream that fails after its headers with a retryable error is reattached at
once, with no wait and no retry counted, where PySpark raises the error inside
its {{Retrying}}, which waits and counts it (client.rs:2229-2236 at 125654b).
Known difference, not part of this issue: the other defaults of
{{RetryPolicy::default()}} (initial backoff, multiplier, maximum backoff,
jitter) differ from PySpark's {{DefaultPolicy}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]