gabotechs commented on PR #12084: URL: https://github.com/apache/arrow-rs/pull/12084#issuecomment-6659215816
## Benchmark results Ran \`cargo bench -p arrow-ipc --bench ipc_writer --features lz4,zstd -- lz4\` on this machine (16 cores), comparing the base of this stack (#12083, no buffer reuse) against this commit (with buffer reuse). Criterion's median times: | Benchmark | Before | After | Change | |---|---|---|---| | `StreamWriter/write_10/lz4` | 2.4401 ms | 2.3848 ms | **-2.26%** | | `StreamEncoder/encode_10/lz4` (single buffer, 8192 rows) | 2.3916 ms | 2.3666 ms | -1.04% (noise) | | `StreamEncoder/encode_10/lz4/rows=8192/threads=1` | 2.4813 ms | 2.4213 ms | **-2.42%** | | `StreamEncoder/encode_10/lz4/rows=65536/threads=1` | 19.977 ms | 20.160 ms | +0.91% (noise) | | `StreamEncoder/encode_10/lz4/rows=8192/threads=16` | 5.9074 ms | 4.3151 ms | **-26.96%** | | `StreamEncoder/encode_10/lz4/rows=65536/threads=16` | 36.330 ms | 33.958 ms | **-6.53%** | Single-threaded cases improve modestly (~1-2%, within noise for the larger 65536-row buffer where compression itself dominates) since there's just one fewer allocation per write. The clear win is under concurrency: with 16 threads each encoding independently, avoiding the lz4 frame encoder's hash table/block buffer allocation on every `compress_lz4` call cuts allocator contention, giving ~27% at 8192 rows and ~6.5% at 65536 rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
