Tim Meehan created SPARK-61412:
----------------------------------
Summary: Send the user agent as client_type on the execute,
analyze, config, interrupt, release and error-detail requests, as PySpark does
Key: SPARK-61412
URL: https://issues.apache.org/jira/browse/SPARK-61412
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
The Rust client sends no {{client_type}} on ExecutePlan, AnalyzePlan, Config,
Interrupt, ReleaseSession and ReleaseExecute, and the public
fetch_error_details sends none unless the caller set it. PySpark sets
{{client_type}} to the user agent on all of these (and on ReattachExecute and
ReleaseExecute, which copy it from the ExecutePlan request they belong to). A
server, proxy or audit log that reads {{client_type}} sees PySpark's requests
with a client type and the Python client of spark-connect-rust's without one.
An ExecutePlan hook also sees the field unset.
{code:python}
# proxy is a loopback gRPC proxy to the server that prints client_type of each
request it forwards.
spark =
SparkSession.builder.remote(f"sc://127.0.0.1:{proxy_port}").getOrCreate()
spark.range(3).collect()
spark.range(3).schema
spark.conf.set("k", "v")
spark.interruptAll()
spark.stop()
{code}
The client_type of each request that the proxy saw, against an OSS Spark 4.3.0
Connect server on loopback:
* pyspark-client 4.3.0: ExecutePlan, AnalyzePlan, Config, Interrupt,
ReleaseExecute and ReleaseSession each carry {{_SPARK_CONNECT_PYTHON
spark/4.3.0 os/linux python/3.11.13}}.
* The Python client at spark-connect-rust master 125654b: the same six requests
carry no {{client_type}} (the field is absent on the wire).
>From source, at 125654b, crates/spark-connect-core/src/client.rs: execute_plan
>(970), execute_plan_reattachable (1172), analyze_plan (1104), config (1359),
>interrupt (1385), conf_while_raising (1327, the Config read while an error is
>raised) and release_session (1742) never set it, and the public
>fetch_error_details (1249) does not fill it. ReattachExecute and
>ReleaseExecute copy the field from an ExecutePlan request that has none
>(crates/spark-connect-core/src/reattach.rs:113, 163).
Known differences that are out of scope here:
* The user agent itself differs: the Rust client ends it with {{python/rust}}
(crates/spark-connect-core/src/channel.rs:294), PySpark with
{{python/<interpreter version>}}.
* CloneSession: PySpark sends {{client_type="python"}}; the Rust client sends
the user agent (client.rs:1704).
* AddArtifacts and ArtifactStatuses: PySpark sends no {{client_type}}; the Rust
client sends the user agent (client.rs:1932, 1975).
* GetStatus is not implemented in the Rust client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]