fallintoplace opened a new pull request, #2006: URL: https://github.com/apache/iceberg-go/pull/2006
## Summary **Use Parquet dictionaries during row-group pruning.** - **EqualTo and IN** predicates can skip dictionary-only row groups. - Keeps the current stats and Bloom filter checks. - Runs the dictionary check between stats and Bloom filters. - Only trusts a dictionary when encoding stats show no plain fallback pages. - Keeps the row group for missing metadata, unsupported types, malformed pages, or read errors. ## Why Min and max stats can be too broad for low-cardinality columns. A complete dictionary can prove that none of the requested values exist, so the row group does not need to be read. ## Tests - `GOMAXPROCS=2 go test -p 1 -vet=off ./table/internal ./table -count=1` - Focused dictionary, Bloom filter, scan-planning, and malformed-page tests ## Benchmark Command: `GOMAXPROCS=2 go test -p 1 -vet=off ./table/internal -run '^$' -bench '^BenchmarkDictionaryRowGroupPruning$' -benchtime=1s -count=3` Workload: 16 row groups with 4,096 low-cardinality INT32 values each. The queried value is absent from every dictionary. - **Without dictionary:** 1.38 to 1.61 ms/op, about 1.62 MB/op. - **With dictionary:** 0.65 to 0.81 ms/op, about 0.63 MB/op. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
