fallintoplace opened a new pull request, #1164: URL: https://github.com/apache/arrow-go/pull/1164
## Summary - track dictionary entries referenced by encoded values - populate Bloom filters from those entries once per column chunk - cover direct Arrow dictionary arrays, multiple data pages, null indices, and dictionary fallback - keep the existing per-value path after fallback to plain encoding ## Why Bloom filter updates currently happen in the ordinary value-writing paths. Direct dictionary-array writes bypass those paths and write indices directly, which leaves their Bloom filters empty. This can produce false negatives when readers use the filter. Ordinary dictionary encoding also hashes every logical value even though inserting the same hash repeatedly does not change the filter. Reusing the dictionary reduces hashing from the row count to the number of referenced dictionary entries while dictionary encoding remains active. ## Benchmark Apple M1 Pro, 100,000 int32 values, median of 10 runs with a 500 ms benchmark time: | Cardinality | Before | After | Time | B/op | allocs/op | |---:|---:|---:|---:|---:|---:| | 1 | 3.568 ms | 1.745 ms | -51.1% | -75.0% | -51.8% | | 10 | 3.902 ms | 2.136 ms | -45.3% | -71.7% | -50.8% | | 100 | 3.956 ms | 2.118 ms | -46.5% | -69.5% | -49.0% | Higher-cardinality cases that trigger dictionary fallback remain close to the existing path. ## Tests - full repository test suite - race tests for parquet/file, parquet/internal/encoding, and parquet/pqarrow - vet for the changed packages -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
