[
https://issues.apache.org/jira/browse/SPARK-60062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60062:
-----------------------------------
Labels: pull-request-available (was: )
> Allow recreating an expired Spark Connect session via
> ReleaseSession(allow_reconnect=true)
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-60062
> URL: https://issues.apache.org/jira/browse/SPARK-60062
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 4.2.0
> Reporter: Venkata krishnan Sowrirajan
> Priority: Major
> Labels: pull-request-available
>
> When a Spark Connect session is closed, for example by inactivity expiry, its
> session id is tombstoned in `closedSessionsCache`. Any later attempt to use
> that id fails with `INVALID_HANDLE.SESSION_CLOSED`. The tombstone stays until
> it's evicted from the size-bounded cache (
> spark.connect.session.manager.closedSessionsTombstonesSize , default 1000),
> which can take an arbitrarily long time on a quiet server.
> Some clients derive a stable session id from a user/application identity so
> that notebooks can reattach after a kernel restart. Once that session
> expires, they can't knowingly start over under the same id. Calling
> `ReleaseSession(allow_reconnect=true)` on the expired session does nothing,
> because the session is no longer active. The only workaround is to switch to
> a different session id.
> The tombstone exists to stop accidental reuse of a closed session. A client
> that explicitly accepts the state loss should be able to recreate it.
> Proposal:
> • If the session is already closed, `ReleaseSession` with
> `allow_reconnect=true` removes the tombstone, so the same session id creates
> a new, empty session.
> • No proto field changes; the existing field's documentation is extended.
> • Expose `allow_reconnect` in Python `SparkConnectClient.release_session` and
> Scala `SparkConnectClient.releaseSession`.
> • Default behaviour is unchanged.
> Example:
> {code:java}
> spark.client.release_session(allow_reconnect=True)
> spark =
> SparkSession.builder.remote(f"sc://host:15002/;session_id={same_id}").create()
> # new, empty session{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
