kishor-rkrishnan opened a new pull request, #58085:
URL: https://github.com/apache/spark/pull/58085
### What changes were proposed in this pull request?
`CachedRemoteRelation.__del__` (`python/pyspark/sql/connect/plan.py`)
releases a server-cached
DataFrame by issuing a blocking, non-reattachable `ExecutePlan` RPC (a
`RemoveRemoteCachedRelation`
command) through a raw `unary_unary` channel, with **no client-side
deadline**. This PR bounds that
call:
- Add a `release_relation` field to `RpcDeadlines`
(`python/pyspark/sql/connect/client/core.py`),
default `60` (seconds), and include it in `RpcDeadlines.disabled()`.
- Pass it as the gRPC `timeout` on the release call in `plan.py`.
- Document in the `RpcDeadlines` docstring why this non-reattachable
`ExecutePlan` call *is* given a
deadline (unlike a query `ExecutePlan`): a timeout here abandons only a
best-effort cache eviction
that the server performs independently, so nothing recoverable is killed.
No wire-protocol change; the change is entirely client-local.
`DefaultPolicy.can_retry` already
returns `False` for `DEADLINE_EXCEEDED` on non-reattachable RPCs, so on
timeout the release call is
not retried — it raises, is caught by the existing `except Exception`,
logged, and the finalizer
proceeds.
### Why are the changes needed?
Without a deadline this finalizer can block forever if the release response
is never delivered. On
the foreachBatch Spark Connect path it runs on the per-batch critical path:
the Python worker
releases the batch DataFrame (firing this finalizer) *before* it writes its
completion signal, so a
blocked release means the signal is never sent and the driver JVM
(`StreamingForeachBatchHelper`)
stays blocked on its per-batch `dataIn.readInt()` — which also has no
timeout. The streaming query
then stalls indefinitely with no error surfaced (observed as multi-hour
hangs), recovering only on
a manual restart. A bounded deadline converts this indefinite hang into a
bounded wait after which
the worker proceeds.
### Does this PR introduce _any_ user-facing change?
Yes — a behavior change, defaulted to preserve safety:
- The `RemoveRemoteCachedRelation` release RPC now has a default client-side
deadline of 60s (via
the new `RpcDeadlines.release_relation`). Previously it had no deadline
and could block forever if
the response was never delivered.
- On timeout, the eviction is abandoned (the existing warning is logged) and
execution continues.
The eviction is best effort and is also performed server-side, so this
does not leak cached state
in normal operation (and on the foreachBatch path the JVM `finally` evicts
the same relation id).
- Configurable/opt-out via `RpcDeadlines(release_relation=<seconds>)` or
`RpcDeadlines.disabled()`.
### How was this patch tested?
- New unit test
`pyspark.sql.tests.connect.client.test_client.SparkConnectClientTestCase.test_remove_cached_relation_uses_release_relation_deadline`:
triggers `CachedRemoteRelation.__del__` against a captured channel and
asserts (a) the configured
`release_relation` deadline is forwarded as the gRPC `timeout`, and (b)
`RpcDeadlines.disabled()`
forwards `None`.
- Existing `test_each_rpc_receives_configured_deadline` still passes,
confirming the `disabled()`
and per-RPC forwarding paths remain consistent with the new field.
- Standalone end-to-end reproduction (no cluster/build): an in-process
`SparkConnectService` whose
`ExecutePlan` never responds, with a real `SparkConnectClient` pointed at
it. With no deadline the
finalizer blocks indefinitely; with a `release_relation` deadline it
returns within the deadline
(returning in exactly one deadline interval — no retry, since
`DEADLINE_EXCEEDED` is not retried
on this path).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
