tisonkun opened a new pull request, #229: URL: https://github.com/apache/datasketches-rust/pull/229
## Summary - remove the private `CompressedState` and `UncompressedState` representations - encode CPC bitstreams directly into the final `SketchBytes` buffer, then backfill their word counts in the preamble - decode CPC bitstreams directly from borrowed input payload slices into the table and window owned by the resulting sketch - preserve the existing fallible decoder, validation boundaries, and serialized format ## Why The state split was introduced with CPC serde in #84 and follows the Java/C++ CPC implementations, where `CompressedState` / `compressed_state` is a concrete intermediate representation used by compression APIs and characterization code. In this Rust implementation the types are private, are constructed only by serde, and live only long enough to copy payload words between buffers. This keeps the temporary data that the compression algorithm actually needs, such as sorted pair arrays and the bit buffer, but removes the one-use semantic state layer around them. Serialization treats the in-memory sketch as internally valid; deserialization validates untrusted metadata before allocating or decoding and keeps the bounded, fallible bit reader from #227. The resulting change removes 603 lines and adds 358 lines (net -245), including the changelog entry. ## Performance probe A temporary Divan probe (not included in this PR) compared this branch with `main` on the same machine for `lg_k = 10` sketches: | operation | coupons | median (`main` -> this PR) | allocations (`main` -> this PR) | | --- | ---: | ---: | ---: | | serialize | 200 | 1.092 us -> 1.013 us | 3 -> 2 | | serialize | 8,000 | 1.500 us -> 0.822 us | 4 -> 2 | | deserialize | 200 | 1.390 us -> 1.112 us | 4 -> 3 | | deserialize | 8,000 | 3.296 us -> 2.737 us | 5 -> 3 | These are small local microbenchmarks rather than a stable performance contract, but they confirm that removing the payload copies does not introduce a regression and materially reduces allocation count. ## Compatibility - Rust-generated fixtures covering every CPC flavor round-trip byte-for-byte. - Java, C++, and Go serialization snapshots deserialize and reserialize byte-for-byte unchanged. - Truncated streams and targeted corruptions continue to return errors instead of panicking. ## Validation - `cargo x check` - `cargo x test` - `cargo x lint` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
