CurtHagenlocher opened a new pull request, #463: URL: https://github.com/apache/arrow-dotnet/pull/463
## What's Changed `ArrayDataConcatenator` built each output buffer by appending elements one at a time to a growing managed `ArrowBuffer.Builder` and then copying the result into memory from the allocator. This change: - **Writes directly into allocator memory.** Each output buffer is sized up front, allocated from the caller's `MemoryAllocator` (or the default), and filled in place. Allocators aren't required to return zeroed memory, so padding and the unused trailing bits of bitmaps are cleared explicitly. - **Copies unaligned bitmaps in bulk.** Bitmaps that don't start on a byte boundary (e.g. from sliced inputs) used to be copied one bit at a time. The new internal `BitUtility.CopyBits` copies eight bytes at a time on little-endian platforms and falls back to byte-at-a-time otherwise. Run-end-encoded arrays still use builders, because the number of runs isn't known up front. ### Tests - `BitUtility.CopyBits`: every source/destination start position from 0 to 15 with lengths from 0 to 150, including checks that bits outside the copied range are left unchanged. - A new `GarbageMemoryAllocator` test allocator fills new memory with a non-zero pattern and counts outstanding allocations. The existing concatenation test cases, sliced and unsliced, run through it to check the results, that the provided allocator is used, and that nothing leaks. - Checks that padding is zeroed, and a test that concatenates many randomly sliced boolean arrays with nulls. ### Benchmark Adds `ArrayConcatenatorBenchmark`: 10 inputs × 100,000 rows each. "Sliced" slices every input at an odd offset. | Benchmark | Before | After | Speedup | Managed memory before → after | |---|---|---|---|---| | Int32 | 2.19 ms | 1.05 ms | 2.1× | 4.0 MB → ~1 KB | | Int32, sliced | 5.03 ms | 1.09 ms | 4.6× | 4.0 MB → ~1 KB | | Boolean | 154 µs | 8.9 µs | 17× | 245 KB → ~0.8 KB | | Boolean, sliced | 5,682 µs | 19.7 µs | 289× | 245 KB → ~0.8 KB | | String | 16.2 ms | 6.06 ms | 2.7× | 49 MB → ~1.4 KB | | String, sliced | 18.7 ms | 6.06 ms | 3.1× | 49 MB → ~1.4 KB | | StringView | 8.99 ms | 5.18 ms | 1.7× | 15.7 MB → ~1.6 KB | | StringView, sliced | 11.9 ms | 5.23 ms | 2.3× | 15.7 MB → ~1.6 KB | | List | 6.70 ms | 3.39 ms | 2.0× | 10.7 MB → ~1.8 KB | | List, sliced | 9.94 ms | 3.46 ms | 2.9× | 10.7 MB → ~2.4 KB | | DenseUnion | 6.49 ms | 3.06 ms | 2.1× | 10.3 MB → ~2.9 KB | | DenseUnion, sliced | 6.50 ms | 3.11 ms | 2.1× | 10.3 MB → ~2.9 KB | BenchmarkDotNet 0.15.8, .NET 8, Windows 11. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
