wgtmac opened a new pull request, #55111: URL: https://github.com/apache/arrow/pull/55111
### Rationale for this change **Draft reference snapshot — DO NOT REVIEW OR MERGE.** Preserve a local exploration of Parquet compact-Thrift parsing for future investigation. This is not a complete feature implementation or a production-ready parser replacement. The starting point was the Arrow Rust [Parquet metadata article](https://arrow.apache.org/blog/2025/10/23/rust-parquet-metadata/). Its reported up-to-9x results are motivation, not a reproduced C++ result. ### What changes are included in this PR? - Custom compact readers for footer metadata, page headers, column/offset indexes and bloom-filter headers. - A borrowed direct-footer representation, retained buffer ownership, direct schema-tree construction and on-demand compatibility materialization. - Experimental statistics-selection policies and an encoding-presence mask. - An additive `FastFileMetaData` summary API that omits column metadata and schema-tree construction. Its timing is reported separately because it does less work. - An anonymized experiment record and a standalone synthetic-footer benchmark driver. No source dataset, source identifiers, schema contents or local paths are included. See [`THRIFT_CUSTOM_PARSER.md`](https://github.com/wgtmac/arrow/blob/acab062dd31cd6070dcb6f75107f7a9fe84adee3/cpp/src/parquet/THRIFT_CUSTOM_PARSER.md) for the design directions, reproducible commands, discarded experiment, uncompleted ideas and known gaps. On October 23, 2026, seven-sample medians for construction from an already resident footer were: | Workload | Generated API | Direct API, full statistics | Summary-only API | | --- | ---: | ---: | ---: | | Synthetic: 100,000 String columns, 20 row groups, 170,453,566-byte footer | 459.995 ms | 285.512 ms (1.61x) | 187.604 ms (2.45x) | | Workload A: 536,809,024-byte file, 154,661,319-byte footer, 83,925 columns, 8 row groups | 260.994 ms | 163.663 ms (1.59x) | 102.205 ms (2.55x) | Generation/I/O and destruction are excluded from these construction measurements; destruction and statistics-policy results are reported separately in the document. Workload A is a supplemental local observation; public reproduction uses the synthetic generator. These are exploratory measurements, not a controlled attribution of each individual optimization or an end-to-end read benchmark. ### Are these changes tested? Rerun on October 23, 2026, against baseline `ca5564e007a209e251b9d4648af4bc6a53c68c4f` plus this snapshot: - Configured Release build: `make -j8` passed. - Parquet internals: 354 passed, including statistics-policy/mask tests and summary count, buffer-lifetime and invalid-buffer checks. - Parquet reader: 154 passed, 4 skipped. - Arrow reader/writer: 847 passed, 7 skipped; 1 disabled. - Schema: 43 passed. File deserialization: 33 passed. - Standalone benchmark: synthetic footer and anonymous local workload; counts checked across modes. The configuration does not exercise every encryption/compression path. Passing these checks does not establish complete semantic compatibility. Equality, parser limits, ownership/thread-safety, policy forwarding, serialization/fallback behavior, Meson integration and portability remain incomplete or require further audit. PMR, resumable decoding, mmap/cache, SIMD and full metadata projection were discussed but were not established as completed A/B experiments. ### Are there any user-facing changes? Yes. In this experimental snapshot eligible plaintext footers use the direct path by default, `ReaderProperties` has experimental policies, and `FastFileMetaData` exposes summary-only access. `PARQUET_USE_GENERATED_THRIFT=1` selects the generated footer baseline for comparison. Existing entry points remain present, but this PR makes no production API/ABI or behavioral-compatibility claim. ### Was AI used for this PR? AI generated the experimental implementation, benchmark driver, tests and documentation under human direction. AI assisted with scope/privacy checks and validation; this is not a completed human correctness review. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [ ] Human - [x] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
