james-willis opened a new pull request, #100: URL: https://github.com/apache/spark-connect-rust/pull/100
### What changes were proposed in this pull request? Declare `#[pymodule(gil_used = false)]` on the `_pyspark` module in `crates/pyspark-rs/src/lib.rs`, opting the `pyspark-client-rust` extension into free-threaded (no-GIL) CPython. This is the first of the two steps identified in [SPARK-59432](https://issues.apache.org/jira/browse/SPARK-59432). The second — adding a free-threaded (`cp313t`/`cp314t`) wheel build leg — is a separate build/packaging change (abi3 and the free-threaded ABI are mutually exclusive) and is left as a follow-up. ### Why are the changes needed? On a free-threaded interpreter (CPython 3.13t/3.14t, PEP 703), CPython re-enables the GIL process-wide when it imports a PyO3 extension module that has not declared `gil_used = false`. A pure-Rust (tonic) Spark Connect client is well positioned to run with the GIL disabled — it pulls in no `grpcio`, which otherwise re-enables the GIL on import — so this declaration is a prerequisite for the extension to run GIL-off. The declaration is truthful: the extension is already thread-safe. - The global tokio runtime is a `OnceLock<Runtime>` (multi-threaded), and every blocking call releases the GIL via `py.detach(|| block_on(...))`. - Process-global state is behind `OnceLock` / `Mutex` / atomics (e.g. the active-session `OnceLock<Mutex<Option<SparkSession>>>`). - Every `#[pyclass]` is `Sync` (PyO3's free-threaded requirement, enforced at compile time): the core `SparkSession` is built from `Arc` / `Arc<Mutex<..>>` / atomics, stored `Py<T>` handles are `Send + Sync`, and the response stream's `tonic::Streaming` is `Sync` via `sync_wrapper::SyncWrapper`. - There are no `Rc` / `RefCell` / `Cell` / `static mut` in the extension. Behavioral note (not a soundness issue): under free-threading, two Python threads invoking a `&mut self` method on the *same* object instance get a `RuntimeError: Already borrowed` rather than corrupting state, which is the documented PyO3 free-threaded contract. ### Does this PR introduce _any_ user-facing change? No. The attribute is a no-op on GIL-enabled interpreters. It has no runtime effect until a free-threaded wheel is published — the current wheels are `cp39-abi3`, which are not selectable on a free-threaded interpreter — and that wheel build leg is the tracked follow-up. ### How was this patch tested? Existing CI (`cargo fmt --all --check` and the workspace build); the attribute is inert on GIL-enabled builds and compiles under the current pyo3 0.28. End-to-end verification that the GIL is actually disabled (`sys._is_gil_enabled() is False`) will be added together with the free-threaded wheel build leg in the follow-up. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
