Vinod KC created SPARK-58162:
--------------------------------
Summary: Spark Connect client-side unbounded RetryException retry
loop in reattach path can hang forever
Key: SPARK-58162
URL: https://issues.apache.org/jira/browse/SPARK-58162
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.2.0, 4.1.0, 4.3.0, 5.0.0
Reporter: Vinod KC
This is a Spark Connect client-side issue . The bug is entirely in the client's
local retry/reattach handling logic (both the Scala and Python Connect clients).
Spark Connect's client-side reattach logic retries an internal error marker
(RetryException) with no limit — no retry count cap, no backoff, and no time
bound.
This marker is raised client-side when a reattach attempt keeps hitting
`DEADLINE_EXCEEDED`, or when the initial query
never reached the server at all.
If the connection to the server becomes unable to make progress (e.g. a load
balancer or NAT gateway silently drops it, or gRPC keepalive is disabled or
hasn't detected the dead connection yet), the client retries this cycle
forever, entirely on its own — the server is stateless in this exchange and
does nothing wrong. The client never returns and never throws — a permanent
hang, affecting both the Scala and Python clients identically.
A minimal, runnable reproduction is attached:
`spark-connect-retryexception-hang-demo.zip` a real Spark Connect server, a
separate client process, and a small proxy that can be frozen to simulate a
dropped connection. A thread dump confirms the client is permanently stuck
retrying inside its own reattach retry loop, with no way out, while the server
remains idle and unaffected throughout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
