Aurantium-47 opened a new pull request, #55673: URL: https://github.com/apache/arrow/pull/55673
### Rationale for this change Related to #34319; separate from attribute-restoration PR #55668. [@thisisnic's analysis in #34319](https://github.com/apache/arrow/issues/34319#issuecomment-5822219231) already identified deserializing and sanitizing the full R metadata as the main cost when selecting only a few columns from a wide file. That comment explicitly discloses AI-assisted analysis. This draft investigates that first work item; it does not claim to discover the bottleneck or bypass metadata checks for unselected columns. The trade-off is material: the synthetic narrow-projection benchmark improves by about **2.9×**, while metadata that reaches a late fallback becomes about **22% slower** in the sanitizer-only benchmark. This draft is for review of that trade-off and the safety boundary, not a claim that every metadata workload improves. ### What changes are included in this PR? - Check the entire metadata tree for a deliberately narrow safe shape: plain lists, optional plain character names, and unattributed allowed atomic/NULL leaves. - Return that tree unchanged only after all of it passes. Check list-element types before extraction, including promises. - Retain the original filtering and warning/error path for other types, custom attributes/list subclasses, or preflight depth above 64. No deserialization, ASCII, gzip or promise-entry gate is removed. - Preserve `on_save` argument evaluation and avoid dispatching list class methods during preflight. - Add regression coverage for names/empty elements, unsafe warnings and their ordering, malformed matrix error context, custom attributes/deep trees, serialized promises, class method dispatch and delayed `on_save` evaluation. The original protection in #41969 and list-subclass handling in #43895 remain in place. There is no API or file-format change. ### Performance, including the regression Linux x86_64, R 4.5.3, conda-forge Arrow 25.0.1 native library. Synthetic 16,000-column × 10-row Parquet file, with `text` and `levels` attributes on each column. Original attribute restoration is held fixed; this is not a measurement of #55668 combined with this patch. Six alternating trials per end-to-end case, garbage collection before each trial, output equality checked. | Workload | Original median | Candidate median | Result | | --- | ---: | ---: | --- | | Read 20 of 16,000 columns | 1.9650 s | 0.6745 s | 2.91× faster | | Read 200 of 16,000 columns | 1.8195 s | 0.6225 s | 2.92× faster | | Sanitize plain 16,000-column metadata | 1.465 s | 0.277 s | 5.29× faster | | Sanitize with a complex attribute in the final column | 1.406 s | 1.718 s | **22.2% slower** | The last two rows use three trials. A late fallback pays for both the preflight and the original walk. Batched, warmed small-input tests likewise show fallback regressions of about 9%, 15% and 21% at 1, 20 and 200 columns respectively. These are synthetic examples, not a general Arrow speedup claim. ASCII metadata decode plus sanitization measured 1.835 → 0.514 s; gzip-wrapped decode plus sanitization measured 1.727 → 0.452 s (three trials each, identical results). These are metadata-only measurements, not full compressed-Parquet reads. Recorded allocations for the wide sanitizer decreased from 640,288 to 384,192 bytes on the plain path but increased to 1,024,480 bytes on the fallback path. Whole-process peak RSS did not improve: approximately 167 → 168 MiB on the plain path and 167 → 170 MiB on fallback, one isolated process per condition. No memory improvement is claimed. The timing series preceded the final fixes for rejecting custom classes without calling `names(x)` and preserving `on_save` evaluation; the extended sanitizer/compression series includes the former fix. Correctness and the peak-RSS measurements were rerun on the final code. These historical timings should be reconfirmed by CI/review rather than treated as final-code release benchmarks. ### Are these changes tested? Passed locally on the final candidate: - 2,092 original/candidate comparisons of returned values, warning text/order and error class/message, including random nested fixtures, unsafe options, custom attributes, ALTREP sequences and depths through 500 under default limits. - Nested and top-level serialized promise checks in safe/unsafe modes; the harmless `stop()` fixture was not executed. - 105 existing expectations from the working Arrow checkout's metadata tests plus 19 new assertions, with zero failures/errors/skips. The candidate was applied in a separate exported checkout and its R source loaded against the installed Arrow 25.0.1 runtime. - Patch application and whitespace checks against the freshly fetched main `r/R/metadata.R`. Review caught and corrected three initial problems: replacing recursive `purrr::map()` changed indexed error context; the first guard could invoke `names()` methods on a classed list; and the initial early return skipped `on_save` evaluation. The first design was discarded; the latter two have explicit regressions. Not yet validated: - A full current-main native/source package build, full package tests or `R CMD check`. - Older supported R versions. Only R 4.5.3 was exercised; R 4.4 environment provisioning did not complete. R 4.5.3 itself rejects a top-level promise, so the older-R entry guard is not independently verified here. - Equivalent behavior under deliberately reduced expression/stack limits. A controlled stack-exhaustion experiment was interrupted while the original error path was taking too long; it is not counted as passing. - Real-world frequency/acceptability of the late-fallback regression. ### Was AI used for this PR? AI was used for investigation, implementation, differential review, regression tests, benchmark execution and this description. No human review is claimed. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [ ] Human - [x] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
