DB Tsai created SPARK-61384:
-------------------------------
Summary: Add experimental embedded Rust Spark Connect server with
Spark JVM engine
Key: SPARK-61384
URL: https://issues.apache.org/jira/browse/SPARK-61384
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Add an experimental, opt-in Spark Connect server frontend implemented in Rust
and embedded in a Spark JVM through JNI.
Rust owns the gRPC listener, request decoding, protocol sessions, operation
IDs, response framing and stream lifetime. A direct JNI engine interface
invokes SparkConnectPlanner/Catalyst and Spark execution APIs in the hosting
JVM. The JVM is an execution backend; this does not forward service traffic to
a second Spark Connect listener.
Arrow vector value buffers cross JNI through the Arrow C Data Interface with
explicit Java owner retention. Arrow IPC and gRPC still serialize results.
The initial WIP proposes unpublished server/JNI crates, a Java host/engine
adapter, runnable Rust and official PySpark examples, protocol and
ownership/lifecycle tests, and build/acceptance documentation. Existing default
client builds and client APIs stay unchanged.
This is for trusted local workloads. RPC/query/data-type coverage is limited;
authentication, authorization, TLS, full protocol parity and production
resource governance are outside the prototype. The Java adapter uses internal
Spark APIs and requires acceptance against a compatible assembly. Current local
validation targets Spark 4.2.0-SNAPSHOT revision
250ccca3f2b5f043e20b415adf17bda5d202c60f, Scala 2.13 and JDK 21; the repository
currently tracks protocol 4.3.0. Windows and JNI library unloading are
unsupported.
Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]