mkleen opened a new pull request, #11287: URL: https://github.com/apache/arrow-rs/pull/11287
# Which issue does this PR close? - Part of #9296. - Pre-requisite for https://github.com/apache/arrow-rs/pull/11285 # Rationale for this change Decoding Parquet statistics and converting them to Arrow can take as long as running the query itself. For page pruning, DataFusion's `page_filter.rs` asks `StatisticsConverter` for the data page mins, maxes and null counts of every row group it scans. Each request goes through the fully parsed `ColumnIndexMetaData`, so the column index is decoded into `Vec`s first and then copied into Arrow arrays. I'm planning a [follow-up PR]( https://github.com/apache/arrow-rs/pull/11285) that decodes the stored `ColumnIndex` bytes straight into Arrow arrays, skipping the intermediate `ColumnIndexMetaData`. This is a the baseline benchmark for this pr. # What changes are included in this PR? A new `page_index_benchmark` group in `parquet/benches/arrow_statistics.rs`: - Writes an in-memory file with 20 row groups, 10 rows per data page and page-level statistics, with every 7th value null. - Covers `Int64`, `Utf8`, `Utf8View` and `Decimal128(20, 2)`, each with 2,000 and 10,000 data pages in total. - Reads each row group's raw column index bytes (via `column_index_range()`) and, inside the timed loop, does what DataFusion's page pruning effectively does today: `decode_column_index` → `PageIndexBuilder` → `StatisticsConverter::data_page_{mins,maxes,null_counts,nan_counts}`. # Are these changes tested? No. # Are there any user-facing changes? No. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
