rustyconover opened a new pull request, #560: URL: https://github.com/apache/datasketches-cpp/pull/560
To find the median weight, `reverse_purge_hash_map::purge()` allocates and frees an array of up to `MAX_SAMPLE_SIZE` (1024) values through the sketch's allocator on every purge. A sketch over a long stream of distinct items purges constantly, so that's one allocation per purge. This keeps the sample on the stack instead: `V samples[MAX_SAMPLE_SIZE]`, which is 8 KB for 64-bit weights. A purge then makes no allocation. If 8 KB of stack is more than you'd like for this path, I can change it to a buffer the map keeps for reuse. ### Test Added `frequent items: a purge does not allocate` to `frequent_items_sketch_custom_type_test.cpp`. It uses an allocator that counts `allocate()` calls, starts the map at its maximum size so it only purges, and streams 99,000 distinct items after warming up. Without the change it makes 3,960 allocations, one per purge; with it, none. All frequent-items tests pass. We found this in the [DuckDB DataSketches extension](https://github.com/Query-farm/datasketches). There, sketches allocate from DuckDB's per-query arena, which doesn't take back individual frees, so each purge's array was memory the arena never got back. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
