fallintoplace opened a new pull request, #1218: URL: https://github.com/apache/arrow-go/pull/1218
## Summary - **Batch Bloom filter hashing into 1024-value chunks.** - Insert each chunk directly from the column writers. - Reuse fixed scratch storage for the built-in xxhash path. - Keep the existing Hasher interface. Custom hashers still use Sum64s. - Cover spaced values and boolean bitmap paths. - Regenerate the typed column writer. ## Why Large writes used to build byte-slice and hash arrays for the whole input before updating the filter. This keeps the temporary hashing memory bounded and follows the dictionary Bloom-filter work in #1164. ## Benchmark On an Apple M1 Pro with 100k int32 values: - **Materialized:** 838 KB/op, 3 allocs/op - **Bounded:** 35 KB/op, 2 allocs/op The bounded path was also about 10% faster in the local benchmark. ## Tests - go test ./parquet/metadata - go test ./parquet/file -run writer and Bloom-filter tests - go test -race on the touched metadata and file tests - go test ./... -run ^$ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
