Heads up folks!

ZooKeeper Rust client is on the way …



> On Nov 17, 2026, at 7:51 PM, Andrew Kyle Purtell (Jira) via dev 
> <[email protected]> wrote:
> 
> Andrew Kyle Purtell created ZOOKEEPER-5133:
> ----------------------------------------------
> 
>             Summary: Native Rust client and drop-in C/C++ client replacement
>                 Key: ZOOKEEPER-5133
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5133
>             Project: ZooKeeper
>          Issue Type: New Feature
>            Reporter: Andrew Kyle Purtell
>            Assignee: Andrew Kyle Purtell
> 
> 
> We have developed a new Rust ZooKeeper client and would like to contribute it.
> 
> A single, shared protocol and I/O core produces two deliverables:
> 
> *Drop-in C/C++ Replacement Libraries (`libzookeeper_mt` and 
> `libzookeeper_st`)*: Binary- and source-compatible replacements for the 
> legacy C client libraries. They expose identical C headers, identical symbol 
> tables, and identical wire and callback semantics, diverging only where 
> documented to remediate memory corruption, data races, undefined behavior, or 
> protocol defects identified in the legacy C client implementation.
> 
> *Native Rust Client (`apache-zookeeper` and `apache-zookeeper-recipes`)*: A 
> strongly typed, idiomatic Rust package supporting both asynchronous (via 
> Rust's well known `tokio`) and synchronous execution runtimes, persistent and 
> recursive watches, transaction batching, and higher level coordination 
> primitives such as distributed locks, leader latches, and tree caches that 
> will interoperate in a wire compatible way with Apache Curator.
> 
> The primary objective is the elimination of memory safety and concurrency 
> defect classes endemic to C systems programming, such as buffer overreads on 
> untrusted network frames, use-after-free during concurrent session teardown, 
> lock inversions across reentrant application callbacks, and uninitialized 
> memory exposure. We achieve this without requiring modifications to existing 
> C/C++  ZooKeeper client applications or runtime configurations. Both 
> deliverables integrate TLS natively via `rustls`, removing OpenSSL from the 
> dependency and linkage graph.
> 
> The legacy C client remains intact and untouched in the repository and 
> continues to serve as the default build target unless the Rust core is 
> selected during the build configuration process.
> 
> *Highlights*
> 
> *Safety and Formal Verification:* Across all shipped crates, "unsafe" Rust is 
> forbidden everywhere except a single audited Foreign Function Interface (FFI) 
> module. Compile time ownership and lock exclusion proof checking ensure 
> memory- and lock-safety at compile time, and these assurances are further 
> reinforced with formal verification using loom's concurrency permutation 
> model checker, Miri's strict provenance abstract machine interpretation, and 
> continuous property based fuzzing using cargo-fuzz.
> 
> *Performance Superiority:* In a comprehensive head-to-head evaluation, the 
> Rust client outperforms the legacy C client with a 95% confidence interval in 
> 898 of 953 cases (94.2%), achieves statistical parity in 33 cases (3.5%), and 
> trails in 22 cases (2.3%). Client CPU consumption is lower than the legacy C 
> client in 284 of 294 measured cases (96.6%). The performance of the 
> multithreaded drop-in libzookeeper_mt is better than or at statistical parity 
> with the C client in 99.7% of the test cases. The single threaded drop-in 
> libzookeeper_st improves small operation latency across all cases by up to 
> 20%. Under high concurrency pipelining with a window of 1,000 concurrent 
> requests, throughput scales up to 32x with the drop-in MT library and up to 
> 54x with Rust's Tokio, compared to the legacy C MT library's measured 
> scalability at 1x. TLS session establishment is 27% faster than compared with 
> the C client.
> 
> *Superior Reliability:* In concurrent close stress testing (36,000 
> close-under-load iterations), the Rust drop-in libraries recorded zero lost, 
> duplicate, or stale completions, resolving a long standing teardown race 
> condition in the legacy C library, which often fails this test.
> 
> *Overview*
> 
> At the foundation of the Rust client stack sits a serialization tier, which 
> houses the generated wire data structures and encoding codecs derived 
> directly from the protocol schema. This foundation performs no network 
> transport operations and validates buffer boundaries on every read, using an 
> incremental frame decoder that allocates memory strictly as data arrives from 
> the network to protect against malformed frame headers and exhaustion attacks.
> 
> Directly above serialization lies the protocol core ("IOCore"), designed as a 
> deterministic state machine operating entirely independent of sockets, 
> execution threads, or ambient clocks. By consuming raw byte streams, abstract 
> request representations, and discrete timesteps provided by the driver layer, 
> the state machine produces outbound wire streams, ordered application event 
> queues, and connection deadlines. All mutable session context, including 
> request pipelines, transaction counters, keepalive intervals, server 
> resolution tracking, watch tables, authentication state, and staged 
> connection teardown, is contained exclusively within this single-owner 
> protocol machine.
> 
> The driver tier bridges the IOCore to the operating system network 
> facilities, multiplexed polling engines, wakeup primitives, connectivity 
> probes, and TLS sessions. A common execution pipeline compiles into four 
> distinct runtime environments selected by feature gates: a single threaded 
> runtime matching legacy single-threaded semantics, a multithreaded runtime 
> matching legacy multithreaded semantics, a single threaded runtime driven 
> directly by caller event loops, and a tokio-compatible runtime for async 
> applications.
> 
> For Rust-native applications, a high level client layer provides typed 
> request handling, asynchronous pipelining, watch notification streams, and 
> builder interfaces, complemented by Curator-compatible coordination recipes 
> including distributed mutual exclusion, leader election, and hierarchical 
> tree caching. For C-based applications, a foreign function interface bridges 
> the Rust side to a thin C/C++ compatibility shim, exposing a C ABI identical 
> to today's C client, while ensuring the compatibility bridge introduces no 
> overheads.
> 
> The shared protocol state machine and I/O pipeline execute unified logic 
> across all deliverables without dynamic branching. Strict boundaries are 
> maintained across the compilation units, forbidding unsafe operations 
> throughout the codebase with the exception of the isolated foreign function 
> bridge.
> 
> The build harness includes comprehensive automated verification suites 
> including a deterministic fault-injection framework, scripted mock servers, 
> differential cross-checks against legacy implementations, binary layout 
> probes, and continuous fuzz testing to provide confidence that the Rust 
> client implementation is both as memory- and concurrency-safe as promised and 
> fully compatible with legacy client behavior.
> 
> *Architecture*
> 
> The foundational invariant is that application-defined code never executes 
> while a Rust stack frame remains active on the calling thread. This rule 
> insulates user logic, including asynchronous completion callbacks, watch 
> notifications, authentication completions, custom logging hooks, and security 
> mechanism exchanges, from the IOCore. Isolation is realized through a 
> cooperative step-and-drain protocol. When the outer runtime invokes the core 
> engine, the core processes pending transport and protocol state, populates an 
> outbox queue with notifications, and returns control after unwinding all Rust 
> frames and releasing internal locks. The surrounding compatibility shim then 
> iterates through the outbox in native host frames, delivering results to 
> application callbacks or managing thread coordination. When protocol 
> progression depends on the observable outcome of an application notification, 
> such as a session closure or authentication state inspection initiated from 
> within a callback, the engine halts at a synchronization barrier and yields, 
> employing a corountine/continuation model. Once the outer layer completes 
> event dispatch, execution resumes seamlessly from the saved corountine state.
> 
> The isolation strategy adapts naturally across the various runtime 
> environments. In the multithreaded compatibility library, the two-thread 
> architecture of the legacy client is preserved: one background thread drives 
> the network I/O loop while a separate dispatch thread handles completion and 
> watch events in strict wire order. Synchronous API calls submit asynchronous 
> requests internally and wait on native condition variables outside all Rust 
> frames, populating caller buffers only upon wake-up, while any attempt to 
> issue synchronous requests from the network thread is rejected immediately to 
> eliminate deadlock. Rather than blocking within the engine, thread join or 
> detach commands are yielded to the outer layer, allowing foreign threads, 
> completion callbacks, or network callbacks to terminate cleanly without 
> deadlocking. In single-threaded environments, no background threads are 
> created. The application drives I/O directly from its own event loop, 
> processing connection events, outbound flushes, and inbound frames in 
> lockstep and delivering completions inline at historical synchronization 
> points. In Rust-native environments, requests are validated and enqueued 
> synchronously at call time to ensure submission order matches wire 
> transmission order, routing responses to asynchronous futures, one-shot watch 
> channels, and persistent watch streams with integrated backpressure and 
> consumer lag tracking.
> 
> The drop-in replacement libraries combine a compiled static archive of the 
> bridge crate with a thin compatibility shim and unmodified legacy 
> serialization units. Public data structures remain compiled by the platform C 
> compiler to guarantee exact binary layout and ensure dynamic linking 
> functions without divergence. Callback dispatch routines run entirely within 
> host stack frames, ensuring any non-local jump or exception unwinds only 
> native frames while safely returning unconsumed events to the core. Thread 
> creation, joins, detach operations, and synchronous waits similarly reside in 
> the outer layer so cancellation points never span active core frames. 
> Exported global symbols and Cyrus SASL glue remain isolated within dedicated 
> C units so that external C header details do not leak into the Rust driver.
> 
> *Verification*
> 
> A battery of verification tools guards against regressions, undefined 
> behavior, and concurrency hazards. Abstract machine interpretation under 
> strict pointer provenance rules systematically exercises unit and integration 
> paths to confirm the total absence of dangling references, uninitialized 
> memory accesses, and aliasing violations across the driver core and foreign 
> interface bridge. Concurrency permutation model checking exhaustively 
> explores thread scheduling interleavings across queue implementations, 
> shutdown sequences, and handle lifecycle operations to ensure synchronization 
> freedom. Continuous coverage-guided fuzzing targets serialization formats, 
> incremental frame reconstruction, transaction batches, stream decoders, 
> authentication handshakes, and cryptographic processing pipelines under 
> address and leak sanitization. Structural layout checkers compile native 
> probe programs across all supported build configurations to assert that every 
> bridge type and public structure matches expected target alignment and 
> padding. Specialized memory scans verify that sensitive cryptographic key 
> material and session credentials are wiped from deallocated heap memory. 
> These verification passes complement extensive deterministic simulation 
> suites, property-based tests, and integration test suites spanning thousands 
> of test scenarios.
> 
> This verification strategy eliminates whole classes of memory and concurrency 
> vulnerabilities present in the C client implementation. Invalid or 
> uninitialized client handle references are intercepted at entry boundaries 
> and converted into clean argument rejections rather than triggering 
> null-pointer faults. Buffer operations on data payloads, string collections, 
> and access control lists validate dimensions and capacities before memory 
> copies occur, eliminating heap corruption and buffer overreads. Framing logic 
> guarantees that malformed or truncated network frames result in graceful 
> connection termination rather than parsing beyond allocated packet bounds or 
> exposing uninitialized stack data to user callbacks. Transaction batch 
> failures consistently initialize all caller result slots with defined error 
> codes rather than leaving uninitialized memory in place. Shared process-wide 
> counters and credential buffers are fully encapsulated within isolated 
> per-session states, eliminating unsynchronized cross-thread mutations. 
> Internal locks are never held while invoking application notifications, 
> preventing deadlock when user callbacks make reentrant client calls. Host 
> failover state machines advance monotonically, eliminating cursor progression 
> defects that previously bypassed ensemble nodes during connection retries. 
> Finally, teardown sequences coordinate thread termination and event delivery 
> so that queued completions are guaranteed to reach waiting callers rather 
> than being dropped during shutdown.
> 
> Applications that employ external authentication mechanisms across multiple 
> threads must continue to configure thread synchronization within the 
> third-party library, as those external caches reside outside the client 
> boundary.
> 
> *Compatibility*
> 
> An existing application built against the original C client can link directly 
> against or dynamically load the replacement libraries with no modifications 
> to application code, build configurations, or operational deployments. 
> Complete fidelity is maintained across all observable dimensions of the 
> programming interface: installed public headers are byte-for-byte identical, 
> platform library names and major shared object versions match historical 
> targets, and exported symbol sets maintain exact parity in symbol names, 
> function signatures, data object sizes, and mutability without introducing 
> extraneous symbols or omitting existing ones. Protocol framing, request 
> sequencing, timeout limits, hash-table watcher traversal order, error return 
> codes, operating system error hints, callback delivery sequences, and 
> formatted diagnostic messages similarly reflect legacy behaviors. Runtime 
> execution characteristics are fully preserved, running asynchronous 
> completions and watch events on a dedicated completion thread in 
> multithreaded builds while dispatching them inline during event loop 
> processing in single-threaded builds.
> 
> Symbol visibility boundaries are strictly enforced during linking, ensuring 
> internal helpers and runtime components remain fully unexported while 
> matching the historical public inventory across every build configuration. In 
> every build variant, a fixed subset of the public surface represents exported 
> data objects, split between read-only constants and mutable configuration 
> variables. The legacy build infrastructure requires only minimal adjustments 
> to delegate to the new core.
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v10.3.26#10030026)

Reply via email to