Andrew Kyle Purtell created ZOOKEEPER-5133:
----------------------------------------------
Summary: Native Rust client and drop-in C/C++ client replacement
Key: ZOOKEEPER-5133
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5133
Project: ZooKeeper
Issue Type: New Feature
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
We have developed a new Rust ZooKeeper client and would like to contribute it.
A single, shared protocol and I/O core produces two deliverables:
*Drop-in C/C++ Replacement Libraries (`libzookeeper_mt` and
`libzookeeper_st`)*: Binary- and source-compatible replacements for the legacy
C client libraries. They expose identical C headers, identical symbol tables,
and identical wire and callback semantics, diverging only where documented to
remediate memory corruption, data races, undefined behavior, or protocol
defects identified in the legacy C client implementation.
*Native Rust Client (`apache-zookeeper` and `apache-zookeeper-recipes`)*: A
strongly typed, idiomatic Rust package supporting both asynchronous (via Rust's
well known `tokio`) and synchronous execution runtimes, persistent and
recursive watches, transaction batching, and higher level coordination
primitives such as distributed locks, leader latches, and tree caches that will
interoperate in a wire compatible way with Apache Curator.
The primary objective is the elimination of memory safety and concurrency
defect classes endemic to C systems programming, such as buffer overreads on
untrusted network frames, use-after-free during concurrent session teardown,
lock inversions across reentrant application callbacks, and uninitialized
memory exposure. We achieve this without requiring modifications to existing
C/C++ ZooKeeper client applications or runtime configurations. Both
deliverables integrate TLS natively via `rustls`, removing OpenSSL from the
dependency and linkage graph.
The legacy C client remains intact and untouched in the repository and
continues to serve as the default build target unless the Rust core is selected
during the build configuration process.
*Highlights*
*Safety and Formal Verification:* Across all shipped crates, "unsafe" Rust is
forbidden everywhere except a single audited Foreign Function Interface (FFI)
module. Compile time ownership and lock exclusion proof checking ensure memory-
and lock-safety at compile time, and these assurances are further reinforced
with formal verification using loom's concurrency permutation model checker,
Miri's strict provenance abstract machine interpretation, and continuous
property based fuzzing using cargo-fuzz.
*Performance Superiority:* In a comprehensive head-to-head evaluation, the Rust
client outperforms the legacy C client with a 95% confidence interval in 898 of
953 cases (94.2%), achieves statistical parity in 33 cases (3.5%), and trails
in 22 cases (2.3%). Client CPU consumption is lower than the legacy C client in
284 of 294 measured cases (96.6%). The performance of the multithreaded drop-in
libzookeeper_mt is better than or at statistical parity with the C client in
99.7% of the test cases. The single threaded drop-in libzookeeper_st improves
small operation latency across all cases by up to 20%. Under high concurrency
pipelining with a window of 1,000 concurrent requests, throughput scales up to
32x with the drop-in MT library and up to 54x with Rust's Tokio, compared to
the legacy C MT library's measured scalability at 1x. TLS session establishment
is 27% faster than compared with the C client.
*Superior Reliability:* In concurrent close stress testing (36,000
close-under-load iterations), the Rust drop-in libraries recorded zero lost,
duplicate, or stale completions, resolving a long standing teardown race
condition in the legacy C library, which often fails this test.
*Overview*
At the foundation of the Rust client stack sits a serialization tier, which
houses the generated wire data structures and encoding codecs derived directly
from the protocol schema. This foundation performs no network transport
operations and validates buffer boundaries on every read, using an incremental
frame decoder that allocates memory strictly as data arrives from the network
to protect against malformed frame headers and exhaustion attacks.
Directly above serialization lies the protocol core ("IOCore"), designed as a
deterministic state machine operating entirely independent of sockets,
execution threads, or ambient clocks. By consuming raw byte streams, abstract
request representations, and discrete timesteps provided by the driver layer,
the state machine produces outbound wire streams, ordered application event
queues, and connection deadlines. All mutable session context, including
request pipelines, transaction counters, keepalive intervals, server resolution
tracking, watch tables, authentication state, and staged connection teardown,
is contained exclusively within this single-owner protocol machine.
The driver tier bridges the IOCore to the operating system network facilities,
multiplexed polling engines, wakeup primitives, connectivity probes, and TLS
sessions. A common execution pipeline compiles into four distinct runtime
environments selected by feature gates: a single threaded runtime matching
legacy single-threaded semantics, a multithreaded runtime matching legacy
multithreaded semantics, a single threaded runtime driven directly by caller
event loops, and a tokio-compatible runtime for async applications.
For Rust-native applications, a high level client layer provides typed request
handling, asynchronous pipelining, watch notification streams, and builder
interfaces, complemented by Curator-compatible coordination recipes including
distributed mutual exclusion, leader election, and hierarchical tree caching.
For C-based applications, a foreign function interface bridges the Rust side to
a thin C/C++ compatibility shim, exposing a C ABI identical to today's C
client, while ensuring the compatibility bridge introduces no overheads.
The shared protocol state machine and I/O pipeline execute unified logic across
all deliverables without dynamic branching. Strict boundaries are maintained
across the compilation units, forbidding unsafe operations throughout the
codebase with the exception of the isolated foreign function bridge.
The build harness includes comprehensive automated verification suites
including a deterministic fault-injection framework, scripted mock servers,
differential cross-checks against legacy implementations, binary layout probes,
and continuous fuzz testing to provide confidence that the Rust client
implementation is both as memory- and concurrency-safe as promised and fully
compatible with legacy client behavior.
*Architecture*
The foundational invariant is that application-defined code never executes
while a Rust stack frame remains active on the calling thread. This rule
insulates user logic, including asynchronous completion callbacks, watch
notifications, authentication completions, custom logging hooks, and security
mechanism exchanges, from the IOCore. Isolation is realized through a
cooperative step-and-drain protocol. When the outer runtime invokes the core
engine, the core processes pending transport and protocol state, populates an
outbox queue with notifications, and returns control after unwinding all Rust
frames and releasing internal locks. The surrounding compatibility shim then
iterates through the outbox in native host frames, delivering results to
application callbacks or managing thread coordination. When protocol
progression depends on the observable outcome of an application notification,
such as a session closure or authentication state inspection initiated from
within a callback, the engine halts at a synchronization barrier and yields,
employing a corountine/continuation model. Once the outer layer completes event
dispatch, execution resumes seamlessly from the saved corountine state.
The isolation strategy adapts naturally across the various runtime
environments. In the multithreaded compatibility library, the two-thread
architecture of the legacy client is preserved: one background thread drives
the network I/O loop while a separate dispatch thread handles completion and
watch events in strict wire order. Synchronous API calls submit asynchronous
requests internally and wait on native condition variables outside all Rust
frames, populating caller buffers only upon wake-up, while any attempt to issue
synchronous requests from the network thread is rejected immediately to
eliminate deadlock. Rather than blocking within the engine, thread join or
detach commands are yielded to the outer layer, allowing foreign threads,
completion callbacks, or network callbacks to terminate cleanly without
deadlocking. In single-threaded environments, no background threads are
created. The application drives I/O directly from its own event loop,
processing connection events, outbound flushes, and inbound frames in lockstep
and delivering completions inline at historical synchronization points. In
Rust-native environments, requests are validated and enqueued synchronously at
call time to ensure submission order matches wire transmission order, routing
responses to asynchronous futures, one-shot watch channels, and persistent
watch streams with integrated backpressure and consumer lag tracking.
The drop-in replacement libraries combine a compiled static archive of the
bridge crate with a thin compatibility shim and unmodified legacy serialization
units. Public data structures remain compiled by the platform C compiler to
guarantee exact binary layout and ensure dynamic linking functions without
divergence. Callback dispatch routines run entirely within host stack frames,
ensuring any non-local jump or exception unwinds only native frames while
safely returning unconsumed events to the core. Thread creation, joins, detach
operations, and synchronous waits similarly reside in the outer layer so
cancellation points never span active core frames. Exported global symbols and
Cyrus SASL glue remain isolated within dedicated C units so that external C
header details do not leak into the Rust driver.
*Verification*
A battery of verification tools guards against regressions, undefined behavior,
and concurrency hazards. Abstract machine interpretation under strict pointer
provenance rules systematically exercises unit and integration paths to confirm
the total absence of dangling references, uninitialized memory accesses, and
aliasing violations across the driver core and foreign interface bridge.
Concurrency permutation model checking exhaustively explores thread scheduling
interleavings across queue implementations, shutdown sequences, and handle
lifecycle operations to ensure synchronization freedom. Continuous
coverage-guided fuzzing targets serialization formats, incremental frame
reconstruction, transaction batches, stream decoders, authentication
handshakes, and cryptographic processing pipelines under address and leak
sanitization. Structural layout checkers compile native probe programs across
all supported build configurations to assert that every bridge type and public
structure matches expected target alignment and padding. Specialized memory
scans verify that sensitive cryptographic key material and session credentials
are wiped from deallocated heap memory. These verification passes complement
extensive deterministic simulation suites, property-based tests, and
integration test suites spanning thousands of test scenarios.
This verification strategy eliminates whole classes of memory and concurrency
vulnerabilities present in the C client implementation. Invalid or
uninitialized client handle references are intercepted at entry boundaries and
converted into clean argument rejections rather than triggering null-pointer
faults. Buffer operations on data payloads, string collections, and access
control lists validate dimensions and capacities before memory copies occur,
eliminating heap corruption and buffer overreads. Framing logic guarantees that
malformed or truncated network frames result in graceful connection termination
rather than parsing beyond allocated packet bounds or exposing uninitialized
stack data to user callbacks. Transaction batch failures consistently
initialize all caller result slots with defined error codes rather than leaving
uninitialized memory in place. Shared process-wide counters and credential
buffers are fully encapsulated within isolated per-session states, eliminating
unsynchronized cross-thread mutations. Internal locks are never held while
invoking application notifications, preventing deadlock when user callbacks
make reentrant client calls. Host failover state machines advance
monotonically, eliminating cursor progression defects that previously bypassed
ensemble nodes during connection retries. Finally, teardown sequences
coordinate thread termination and event delivery so that queued completions are
guaranteed to reach waiting callers rather than being dropped during shutdown.
Applications that employ external authentication mechanisms across multiple
threads must continue to configure thread synchronization within the
third-party library, as those external caches reside outside the client
boundary.
*Compatibility*
An existing application built against the original C client can link directly
against or dynamically load the replacement libraries with no modifications to
application code, build configurations, or operational deployments. Complete
fidelity is maintained across all observable dimensions of the programming
interface: installed public headers are byte-for-byte identical, platform
library names and major shared object versions match historical targets, and
exported symbol sets maintain exact parity in symbol names, function
signatures, data object sizes, and mutability without introducing extraneous
symbols or omitting existing ones. Protocol framing, request sequencing,
timeout limits, hash-table watcher traversal order, error return codes,
operating system error hints, callback delivery sequences, and formatted
diagnostic messages similarly reflect legacy behaviors. Runtime execution
characteristics are fully preserved, running asynchronous completions and watch
events on a dedicated completion thread in multithreaded builds while
dispatching them inline during event loop processing in single-threaded builds.
Symbol visibility boundaries are strictly enforced during linking, ensuring
internal helpers and runtime components remain fully unexported while matching
the historical public inventory across every build configuration. In every
build variant, a fixed subset of the public surface represents exported data
objects, split between read-only constants and mutable configuration variables.
The legacy build infrastructure requires only minimal adjustments to delegate
to the new core.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)