hamilton-earthscope opened a new pull request, #573: URL: https://github.com/apache/arrow-go/pull/573
### Rationale for this change This PR implements the same allocation improvements as https://github.com/apache/arrow-go/pull/557 but for the Take list kernel. ### What changes are included in this PR? - Pre-allocate upfront with a better estimate of the necessary buffer size to eliminate repeated reallocations. - Use exponential growth for additional allocations for O(log n) total reallocations. ### Are these changes tested? New benchmarks added for take list kernel ### Are there any user-facing changes? No ### Performance Comparison on `main` ``` goos: darwin goarch: arm64 pkg: github.com/apache/arrow-go/v18/arrow/compute cpu: Apple M3 Max BenchmarkTakeList/SmallBatch_ShortLists-16 2320 555889 ns/op 1798923 rows/sec 3355390 B/op 396 allocs/op BenchmarkTakeList/MediumBatch_ShortLists-16 31 37600434 ns/op 265955 rows/sec 324238694 B/op 3214 allocs/op BenchmarkTakeList/LargeBatch_ShortLists-16 2 653682896 ns/op 76490 rows/sec 7877124660 B/op 15751 allocs/op BenchmarkTakeList/XLargeBatch_ShortLists-16 1 2019873959 ns/op 49508 rows/sec 31380868976 B/op 31489 allocs/op BenchmarkTakeList/SmallBatch_MediumLists-16 334 3435205 ns/op 291104 rows/sec 43059334 B/op 1086 allocs/op BenchmarkTakeList/MediumBatch_MediumLists-16 5 230308533 ns/op 43420 rows/sec 4041679715 B/op 10098 allocs/op BenchmarkTakeList/LargeBatch_MediumLists-16 1 4600489959 ns/op 10868 rows/sec 100213267960 B/op 50328 allocs/op BenchmarkTakeList/XLargeBatch_MediumLists-16 1 16462343792 ns/op 6074 rows/sec 400427784144 B/op 101032 allocs/op BenchmarkTakeList/LargeBatch_ShortLists_Large-16 1 1682293042 ns/op 29721 rows/sec 31379855376 B/op 31422 allocs/op BenchmarkTakeList/XLargeBatch_MediumLists_Large-16 1 33800353917 ns/op 2959 rows/sec 800433389776 B/op 102480 allocs/op BenchmarkTakeListPartitionPattern-16 1 3003105000 ns/op 16649 rows/sec 69581474288 B/op 46479 allocs/op PASS ok github.com/apache/arrow-go/v18/arrow/compute 71.015s ``` on this branch ``` goos: darwin goarch: arm64 pkg: github.com/apache/arrow-go/v18/arrow/compute cpu: Apple M3 Max BenchmarkTakeList/SmallBatch_ShortLists-16 25522 46138 ns/op 21674062 rows/sec 50668 B/op 83 allocs/op BenchmarkTakeList/MediumBatch_ShortLists-16 3792 316046 ns/op 31641022 rows/sec 457558 B/op 83 allocs/op BenchmarkTakeList/LargeBatch_ShortLists-16 804 1521240 ns/op 32867968 rows/sec 2232578 B/op 84 allocs/op BenchmarkTakeList/XLargeBatch_ShortLists-16 416 2832247 ns/op 35307705 rows/sec 4435314 B/op 84 allocs/op BenchmarkTakeList/SmallBatch_MediumLists-16 9444 125321 ns/op 7979591 rows/sec 173603 B/op 83 allocs/op BenchmarkTakeList/MediumBatch_MediumLists-16 1176 999217 ns/op 10007857 rows/sec 1653926 B/op 83 allocs/op BenchmarkTakeList/LargeBatch_MediumLists-16 232 4913249 ns/op 10176590 rows/sec 8229752 B/op 85 allocs/op BenchmarkTakeList/XLargeBatch_MediumLists-16 128 9309120 ns/op 10742230 rows/sec 16428821 B/op 85 allocs/op BenchmarkTakeList/LargeBatch_ShortLists_Large-16 739 1560044 ns/op 32050453 rows/sec 3428837 B/op 84 allocs/op BenchmarkTakeList/XLargeBatch_MediumLists_Large-16 122 9712600 ns/op 10295969 rows/sec 24834215 B/op 85 allocs/op BenchmarkTakeListPartitionPattern-16 96 11756706 ns/op 4252901 rows/sec 18421151 B/op 98 allocs/op PASS ok github.com/apache/arrow-go/v18/arrow/compute 17.869s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
