fallintoplace opened a new pull request, #1403: URL: https://github.com/apache/arrow-go/pull/1403
## What Reuse dictionary checks and index mappings in `array.Concatenate`. ## Why Many chunks share dictionary data. Rechecking values and rebuilding mappings for each chunk adds repeated work. ## Implementation - Skip value comparison when chunks share the same dictionary data. - Reuse transposition buffers for repeated dictionaries. - Add correctness coverage and a public API benchmark. ## Benchmark Public `array.Concatenate` call with 12 chunks and 8,192 dictionary entries. Go 1.26.3, Apple M1 Pro (`darwin/arm64`), `GOMAXPROCS=1`; 10 runs of 200 ms per case. Base `de0c94e` vs head `651fe14`. | Metric | Before | After | Change | | --- | ---: | ---: | ---: | | Shared dictionary, time | 82.374 µs/op | 3.723 µs/op | -95.48% | | Repeated mixed dictionaries, time | 5.593 ms/op | 2.039 ms/op | -63.54% | | Shared dictionary, bytes | 15.42 KiB/op | 14.48 KiB/op | -6.08% | | Repeated mixed dictionaries, bytes | 4.262 MiB/op | 2.784 MiB/op | -34.69% | | Shared dictionary, allocations | 23 allocs/op | 11 allocs/op | -52.17% | | Repeated mixed dictionaries, allocations | 99.04k allocs/op | 25.26k allocs/op | -74.49% | Both time cases improved. The benchmark excludes setup and releases each output. Command: `GOMAXPROCS=1 go test ./arrow/array -run '^$' -bench '^BenchmarkConcatenateDictionaryReuse$' -benchmem -benchtime=200ms -count=10 -cpu=1`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
