venkata91 opened a new pull request, #59277:
URL: https://github.com/apache/spark/pull/59277
### What changes were proposed in this pull request?
This lets a client knowingly recreate a Spark Connect session id whose
session has already been closed, for example after inactivity expiry.
- `SparkConnectSessionManager.closeSession(key, allowReconnect = true)`: if
no active session exists for the key, it now drops the key from
`closedSessionsCache`. Previously this call did nothing. The next
`getOrCreateIsolatedSession` with the same id creates a new, empty session.
- The `ReleaseSessionRequest.allow_reconnect` proto documentation (and the
generated `base_pb2.pyi`) describes the new behaviour. No proto fields change.
- Python `SparkConnectClient.release_session(allow_reconnect=False)` and
Scala `SparkConnectClient.releaseSession(allowReconnect = false)` now expose
the existing field.
The default behaviour is unchanged. Without `allow_reconnect`, a closed
session id still fails with `INVALID_HANDLE.SESSION_CLOSED`.
### Why are the changes needed?
Clients often derive a stable session id from a (user, application) identity
so that notebooks and kernels can reattach to the same server-side session.
Once that session expires
(`spark.connect.session.manager.defaultSessionTimeout`), its id is tombstoned.
Every later attempt to use it fails with `INVALID_HANDLE.SESSION_CLOSED` until
the tombstone is evicted from the count-bounded cache
(`closedSessionsTombstonesSize`, default 1000), which can take an arbitrarily
long time on a quiet server.
Today the only way out is to change the session id: users must rename their
application, or clients must invent random ids. Calling
`ReleaseSession(allow_reconnect=true)` on the expired session does nothing
because the session is no longer active.
The tombstone is there to stop *accidental* reuse of a closed session, and
silently losing state would be surprising. A client that has been told the
session is gone, and explicitly accepts that its state (temp views, temp UDFs,
cached data, current catalog) is lost, should be able to start over under the
same id. `allow_reconnect` already has exactly these documented semantics ("the
server may permit the client to reconnect to the session post-release, even if
the session state has been cleaned").
### Does this PR introduce _any_ user-facing change?
Yes, opt-in only. A client that gets `INVALID_HANDLE.SESSION_CLOSED` can now
do:
```python
spark.client.release_session(allow_reconnect=True)
spark =
SparkSession.builder.remote(f"sc://host:15002/;session_id={same_id}").create()
# new, empty session
```
Before this change, the second call failed with
`INVALID_HANDLE.SESSION_CLOSED`.
A follow-up could add a public builder-level option for this (for example,
recreating an expired session on `create()`). Feedback on the preferred API
shape is welcome.
### How was this patch tested?
- New `SparkConnectSessionManagerSuite` test: an expired session stays
tombstoned after a plain `closeSession`, and becomes recreatable, with a new
server-side session id, after `closeSession(key, allowReconnect = true)`.
`build/sbt "connect/testOnly
org.apache.spark.sql.connect.service.SparkConnectSessionManagerSuite"`: 11
passed.
- New Python client test checks that `allow_reconnect` is sent on
`ReleaseSession`:
`python -m pytest python/pyspark/sql/tests/connect/client/test_client.py
-k release_session`: passed.
- Manually, against a 4.1.1 server with the default tombstone size: after
expiry, `ReleaseSession` (with or without `allow_reconnect`) left the id
rejected with `SESSION_CLOSED`. This is the behaviour this PR changes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
