[
https://issues.apache.org/jira/browse/SPARK-61410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61410.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 517
[https://github.com/apache/spark-connect-rust/pull/517]
> Raise NO_ACTIVE_SESSION from the first call after the server-side session
> changed, as PySpark does
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-61410
> URL: https://issues.apache.org/jira/browse/SPARK-61410
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Assignee: Tim Meehan
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> When the server-side session changes, the first call of the Rust client
> raises the server's INVALID_HANDLE.SESSION_CHANGED (an UnknownException in
> the Python client of spark-connect-rust), and the calls after it raise
> NO_ACTIVE_SESSION. PySpark raises NO_ACTIVE_SESSION from the first call. Only
> the first call's error differs.
> The change is provoked without a restart: the client's observed server-side
> session ID is set to another UUID, so the request carries an ID the server
> does not hold, and the server answers with SESSION_CHANGED, as after a
> restart.
> {code:python}
> import uuid
> from pyspark.sql.connect.session import SparkSession
> spark = SparkSession(connection="sc://localhost:15002")
> spark.range(3).count()
> spark._client._server_session_id = str(uuid.uuid4())
> for _ in range(2):
> try:
> spark.range(3).collect()
> except Exception as e:
> print(type(e).__name__, e.getCondition())
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: first call SparkConnectException NO_ACTIVE_SESSION;
> second call SparkConnectException NO_ACTIVE_SESSION.
> * spark-connect-rust master 125654b: first call UnknownException
> INVALID_HANDLE.SESSION_CHANGED; second call SparkConnectException
> NO_ACTIVE_SESSION.
> From source, at 125654b: SparkConnectClient::grpc_error closes the client
> when it sees INVALID_HANDLE.SESSION_CHANGED
> (crates/spark-connect-core/src/client.rs:705-709). error_from_status then
> fetches the details of the error under its errorId through error_details,
> which does not check that the client is open, and falls back to the bare
> error if the fetch fails (client.rs:1268-1279, 1286-1305). PySpark does the
> same in _handle_rpc_error, but its _fetch_enriched_error goes through _stub,
> which raises NO_ACTIVE_SESSION once the client is closed
> (pyspark/sql/connect/client/core.py:961-969, 2349-2368, 2429-2447). That
> exception is not a grpc.RpcError, so it is not caught and it replaces the
> error of the server.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]