aik-jahoda opened a new pull request, #73: URL: https://github.com/apache/arrow-dotnet/pull/73
## What's Changed ArrowStreamWriter allocates 8k of memory by creating a new array pool. This change introduces a shared buffer instead of pool to reduce the allocations. The array pool is used to rent small arrays (8 bytes) but the pool allocates much bigger arrays (8kb in total) The Array pool has access time overhead for small arrays compared to direct allocation. Results from benchmarks: Old implementation: | Method | BatchLength | ColumnSetCount | Mean | Error | StdDev | Allocated | |----------- |------------ |--------------- |-----------:|----------:|----------:|----------:| | WriteBatch | 10000 | 10 | 6.118 ms | 0.1215 ms | 0.3345 ms | 248.53 KB | | WriteBatch | 10000 | 14 | 9.788 ms | 0.1910 ms | 0.3396 ms | 324.12 KB | | WriteBatch | 300000 | 10 | 119.351 ms | 3.1713 ms | 9.3008 ms | 248.53 KB | | WriteBatch | 300000 | 14 | 136.697 ms | 2.9229 ms | 8.4799 ms | 324.12 KB | New implementation: | Method | BatchLength | ColumnSetCount | Mean | Error | StdDev | Median | Allocated | |----------- |------------ |--------------- |-----------:|----------:|-----------:|-----------:|----------:| | WriteBatch | 10000 | 10 | 5.925 ms | 0.2057 ms | 0.6001 ms | 5.843 ms | 240.64 KB | | WriteBatch | 10000 | 14 | 8.908 ms | 0.2743 ms | 0.8002 ms | 8.778 ms | 316.23 KB | | WriteBatch | 300000 | 10 | 94.835 ms | 1.7872 ms | 3.7699 ms | 93.892 ms | 240.64 KB | | WriteBatch | 300000 | 14 | 147.995 ms | 3.6873 ms | 10.6975 ms | 144.591 ms | 316.23 KB | Closes #41. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
