sunchao opened a new pull request, #11276: URL: https://github.com/apache/arrow-rs/pull/11276
# Which issue does this PR close? Closes #11275. # Rationale for this change The metadata push decoder walks every row group's column metadata to discover the page-index byte range, including when both index policies are `Skip`. If index bytes are missing, each subsequent `try_decode()` repeats the same discovery. This adds work proportional to the number of column chunks even though the metadata and policies have not changed. # What changes are included in this PR? - Return immediately for `Skip`/`Skip`. - Retain one optional range while waiting for index bytes. Each decode attempt still checks buffer availability; policy changes invalidate the range, and decoding or errors consume it. - Extend the incremental test to cover repeated polls and cleared buffers, and compare all three policy setters against a fresh decoder after changing a pending request. # Are these changes tested? Using Rust 1.98.1: - Metadata unit tests: **40 passed** with default features and **49 passed** with `async,encryption`. - Encrypted page-index integration tests: **4 passed**, covering uniform/nonuniform encryption with synchronous and asynchronous readers. - Parquet library/test Clippy with `async,encryption` and `-D warnings` passed. - Workspace formatting, direct formatting of the changed module, and `git diff --check` passed. The configured registry does not yet provide upstream's locked `opendal 0.59.3`. Local tests and Clippy used a temporary lock resolution with `opendal`/`opendal-core 0.59.2 and its resolved dependency graph. The original upstream lockfile is preserved in this PR; there are no dependency or manifest changes. These results do not establish validation with the committed lockfile. No full workspace suite was run. <details> <summary>Focused test commands</summary> ```text cargo +1.98.1 test --locked -p parquet --lib file::metadata:: cargo +1.98.1 test --locked -p parquet --features async,encryption --lib file::metadata:: cargo +1.98.1 test --locked -p parquet --features async,encryption --test encryption test_decrypt_page_index cargo +1.98.1 clippy --locked -p parquet --lib --tests --features async,encryption -- -D warnings ``` The commands above used the temporary local lock resolution described above. </details> ## Focused performance measurement An external scratch harness compared base `74b630b26d9d` with this change using synthetic metadata and identical dependency locks. Release builds used Rust 1.98.1, disabled Parquet default features, and one pinned CPU on a shared AMD EPYC-Milan host. Results are medians of four process medians per revision, with seven samples per process and balanced ABBA/BAAB ordering. | Operation | Column chunks | Base | This change | |---|---:|---:|---:| | Skip/Skip decode | 10 | 77.4 ns | 84.8 ns | | Skip/Skip decode | 1,000 | 765 ns | 84.9 ns | | Skip/Skip decode | 100,000 | 70.17 us | 85.3 ns | | Repeated decode awaiting index bytes | 10 | 60.7 ns | 26.6 ns | | Repeated decode awaiting index bytes | 1,000 | 4.479 us | 26.5 ns | | Repeated decode awaiting index bytes | 100,000 | 1.615 ms | 26.8 ns | Skip/Skip includes decoder construction and allocation, moving the returned metadata into the next iteration without cloning. The ten-chunk case was about **7.3 ns slower (9.5%)**. Repeated-wait measurements exclude the first range discovery and include returning/dropping `NeedsData`; assertions check the exact requested range and unchanged metadata. These measurements exclude footer/index deserialization and storage I/O and do not establish query-time gains. The scratch harness is not added to this PR. # Are there any user-facing changes? No public API, policy-default, requested-range, or decoding-behavior changes. Each metadata decoder retains at most one additional optional byte range. # AI assistance OpenAI Codex assisted with porting the implementation and tests, reviewing the change, running validation and the scratch benchmark, and preparing this description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
