ariesdevil opened a new pull request, #210: URL: https://github.com/apache/datasketches-rust/pull/210
## Summary This PR reduces `TDigestMut` memory usage by storing compressed and unmerged centroids in a single buffer after the first compression. It preserves the compact `Vec<f64>` staging buffer introduced in #209 for small digests. After compression, the layout becomes: `[compressed centroids | unmerged centroids]` This follows the ClickHouse approach while preserving the existing API, compression behavior, and serialization format. ## Changes - Use one centroid buffer after the first compression. - Compact and merge centroids in place. - Release unused staging and merge capacity. - Validate serialized payload lengths before allocating. - Preserve stable merge ordering and byte-for-byte serialization compatibility. ## Benchmarks Measured with `cargo bench --locked -p datasketches --features tdigest --bench tdigest` on an Apple M5 Pro using Rust 1.86.0. Results are medians from three interleaved release runs comparing `main` (`7e578e7`) with this PR (`d84fa16`). | Benchmark | `main` | This PR | Change | |---|---:|---:|---:| | 1-value lifecycle | 45.4 ns | 45.6 ns | +0.6% | | 8-value lifecycle | 85.2 ns | 81.4 ns | -4.4% | | 64-value lifecycle | 729.5 ns | 708.2 ns | -2.9% | | Serialize 512 64-value partial states | 103.5 µs | 89.6 µs | -13.5% | | Deserialize 512 64-value partial states | 103.8 µs | 110.7 µs | +6.6% | | Merge 64 64-value partial states | 71.4 µs | 69.7 µs | -2.3% | Small-digest heap allocation remains unchanged: | Values | `main` | This PR | |---:|---:|---:| | 1 | 96 B | 96 B | | 8 | 352 B | 352 B | | 64 | 3.10 KB | 3.10 KB | Memory improvements for larger states: | Scenario | `main` | This PR | Change | |---|---:|---:|---:| | Peak heap during initial compression | 59.04 KB | 39.36 KB | -33.3% | | Peak heap during unmerged-tail compression | 59.04 KB | 29.05 KB | -50.8% | | Peak heap serializing 512 partial states | 1.077 MB | 815.1 KB | -24.3% | | Retained size after compressing 1,640 values | 46,008 B | 26,328 B | -42.8% | | Retained size after merging two 1,640-value digests | 65,688 B | 32,888 B | -49.9% | | Frozen 1,640-value digest | 32,856 B | 2,872 B | -91.3% | ## Compatibility and Testing - Serialization remained byte-for-byte identical across 37 differential scenarios. - Existing C++, Java, and Go compatibility snapshots pass. - Added tests for stable merge ordering, overfull deserialized buffers, malformed payloads, and retained capacity. - `cargo x check` passed. - `cargo x test` passed. - Clippy, Rustfmt, Rustdoc, Taplo, and Typos passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
