[ 
https://issues.apache.org/jira/browse/SPARK-61410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61410.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 517
[https://github.com/apache/spark-connect-rust/pull/517]

> Raise NO_ACTIVE_SESSION from the first call after the server-side session 
> changed, as PySpark does
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61410
>                 URL: https://issues.apache.org/jira/browse/SPARK-61410
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Tim Meehan
>            Assignee: Tim Meehan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> When the server-side session changes, the first call of the Rust client 
> raises the server's INVALID_HANDLE.SESSION_CHANGED (an UnknownException in 
> the Python client of spark-connect-rust), and the calls after it raise 
> NO_ACTIVE_SESSION. PySpark raises NO_ACTIVE_SESSION from the first call. Only 
> the first call's error differs.
> The change is provoked without a restart: the client's observed server-side 
> session ID is set to another UUID, so the request carries an ID the server 
> does not hold, and the server answers with SESSION_CHANGED, as after a 
> restart.
> {code:python}
> import uuid
> from pyspark.sql.connect.session import SparkSession
> spark = SparkSession(connection="sc://localhost:15002")
> spark.range(3).count()
> spark._client._server_session_id = str(uuid.uuid4())
> for _ in range(2):
>     try:
>         spark.range(3).collect()
>     except Exception as e:
>         print(type(e).__name__, e.getCondition())
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: first call SparkConnectException NO_ACTIVE_SESSION; 
> second call SparkConnectException NO_ACTIVE_SESSION.
> * spark-connect-rust master 125654b: first call UnknownException 
> INVALID_HANDLE.SESSION_CHANGED; second call SparkConnectException 
> NO_ACTIVE_SESSION.
> From source, at 125654b: SparkConnectClient::grpc_error closes the client 
> when it sees INVALID_HANDLE.SESSION_CHANGED 
> (crates/spark-connect-core/src/client.rs:705-709). error_from_status then 
> fetches the details of the error under its errorId through error_details, 
> which does not check that the client is open, and falls back to the bare 
> error if the fetch fails (client.rs:1268-1279, 1286-1305). PySpark does the 
> same in _handle_rpc_error, but its _fetch_enriched_error goes through _stub, 
> which raises NO_ACTIVE_SESSION once the client is closed 
> (pyspark/sql/connect/client/core.py:961-969, 2349-2368, 2429-2447). That 
> exception is not a grpc.RpcError, so it is not caught and it replaces the 
> error of the server.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to