rustyconover opened a new pull request, #561: URL: https://github.com/apache/datasketches-cpp/pull/561
`tdigest`'s constructor reserves room for `2k + fudge` centroids and `BUFFER_MULTIPLIER` times as many buffered values. For `k = 100` with `double` that's about 10 KB per digest before anything is added, and it happens again on every `deserialize()`. This is the same kind of change as #558 for `kll_sketch`. Instead of reserving everything up front: - `update()` grows the buffer geometrically, starting at 8 values and doubling up to its existing limit. - `compress()` and `merge()` grow the centroids before they need them, also up to the existing limit. Results and the serialized format are unchanged. ### Test Added `tdigest custom allocator: buffers grow as they fill` to `tdigest_custom_allocator_test.cpp`, using `test_allocator`: - An empty digest allocates nothing. Before this change it fails at `REQUIRE( test_allocator_total_bytes == 0 )`. - Three values take at most 8 values of buffer. - A digest of 100,000 updates gives the same quantiles as one built with `std::allocator` without the change. All 17 test suites pass under CTest (GCC 14, aarch64). We found this in the [DuckDB DataSketches extension](https://github.com/Query-farm/datasketches). For GROUP BY queries with many groups, DuckDB creates a state for each input row and merges them later, so the up-front reservation multiplied by millions of states. DuckDB made the same change to its own t-digest. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
