Aurantium-47 opened a new pull request, #55673:
URL: https://github.com/apache/arrow/pull/55673

   ### Rationale for this change
   
   Related to #34319; separate from attribute-restoration PR #55668.
   
   [@thisisnic's analysis in 
#34319](https://github.com/apache/arrow/issues/34319#issuecomment-5822219231) 
already identified deserializing and sanitizing the full R metadata as the main 
cost when selecting only a few columns from a wide file. That comment 
explicitly discloses AI-assisted analysis. This draft investigates that first 
work item; it does not claim to discover the bottleneck or bypass metadata 
checks for unselected columns.
   
   The trade-off is material: the synthetic narrow-projection benchmark 
improves by about **2.9×**, while metadata that reaches a late fallback becomes 
about **22% slower** in the sanitizer-only benchmark. This draft is for review 
of that trade-off and the safety boundary, not a claim that every metadata 
workload improves.
   
   ### What changes are included in this PR?
   
   - Check the entire metadata tree for a deliberately narrow safe shape: plain 
lists, optional plain character names, and unattributed allowed atomic/NULL 
leaves.
   - Return that tree unchanged only after all of it passes. Check list-element 
types before extraction, including promises.
   - Retain the original filtering and warning/error path for other types, 
custom attributes/list subclasses, or preflight depth above 64. No 
deserialization, ASCII, gzip or promise-entry gate is removed.
   - Preserve `on_save` argument evaluation and avoid dispatching list class 
methods during preflight.
   - Add regression coverage for names/empty elements, unsafe warnings and 
their ordering, malformed matrix error context, custom attributes/deep trees, 
serialized promises, class method dispatch and delayed `on_save` evaluation.
   
   The original protection in #41969 and list-subclass handling in #43895 
remain in place. There is no API or file-format change.
   
   ### Performance, including the regression
   
   Linux x86_64, R 4.5.3, conda-forge Arrow 25.0.1 native library. Synthetic 
16,000-column × 10-row Parquet file, with `text` and `levels` attributes on 
each column. Original attribute restoration is held fixed; this is not a 
measurement of #55668 combined with this patch. Six alternating trials per 
end-to-end case, garbage collection before each trial, output equality checked.
   
   | Workload | Original median | Candidate median | Result |
   | --- | ---: | ---: | --- |
   | Read 20 of 16,000 columns | 1.9650 s | 0.6745 s | 2.91× faster |
   | Read 200 of 16,000 columns | 1.8195 s | 0.6225 s | 2.92× faster |
   | Sanitize plain 16,000-column metadata | 1.465 s | 0.277 s | 5.29× faster |
   | Sanitize with a complex attribute in the final column | 1.406 s | 1.718 s 
| **22.2% slower** |
   
   The last two rows use three trials. A late fallback pays for both the 
preflight and the original walk. Batched, warmed small-input tests likewise 
show fallback regressions of about 9%, 15% and 21% at 1, 20 and 200 columns 
respectively. These are synthetic examples, not a general Arrow speedup claim.
   
   ASCII metadata decode plus sanitization measured 1.835 → 0.514 s; 
gzip-wrapped decode plus sanitization measured 1.727 → 0.452 s (three trials 
each, identical results). These are metadata-only measurements, not full 
compressed-Parquet reads.
   
   Recorded allocations for the wide sanitizer decreased from 640,288 to 
384,192 bytes on the plain path but increased to 1,024,480 bytes on the 
fallback path. Whole-process peak RSS did not improve: approximately 167 → 168 
MiB on the plain path and 167 → 170 MiB on fallback, one isolated process per 
condition. No memory improvement is claimed.
   
   The timing series preceded the final fixes for rejecting custom classes 
without calling `names(x)` and preserving `on_save` evaluation; the extended 
sanitizer/compression series includes the former fix. Correctness and the 
peak-RSS measurements were rerun on the final code. These historical timings 
should be reconfirmed by CI/review rather than treated as final-code release 
benchmarks.
   
   ### Are these changes tested?
   
   Passed locally on the final candidate:
   
   - 2,092 original/candidate comparisons of returned values, warning 
text/order and error class/message, including random nested fixtures, unsafe 
options, custom attributes, ALTREP sequences and depths through 500 under 
default limits.
   - Nested and top-level serialized promise checks in safe/unsafe modes; the 
harmless `stop()` fixture was not executed.
   - 105 existing expectations from the working Arrow checkout's metadata tests 
plus 19 new assertions, with zero failures/errors/skips. The candidate was 
applied in a separate exported checkout and its R source loaded against the 
installed Arrow 25.0.1 runtime.
   - Patch application and whitespace checks against the freshly fetched main 
`r/R/metadata.R`.
   
   Review caught and corrected three initial problems: replacing recursive 
`purrr::map()` changed indexed error context; the first guard could invoke 
`names()` methods on a classed list; and the initial early return skipped 
`on_save` evaluation. The first design was discarded; the latter two have 
explicit regressions.
   
   Not yet validated:
   
   - A full current-main native/source package build, full package tests or `R 
CMD check`.
   - Older supported R versions. Only R 4.5.3 was exercised; R 4.4 environment 
provisioning did not complete. R 4.5.3 itself rejects a top-level promise, so 
the older-R entry guard is not independently verified here.
   - Equivalent behavior under deliberately reduced expression/stack limits. A 
controlled stack-exhaustion experiment was interrupted while the original error 
path was taking too long; it is not counted as passing.
   - Real-world frequency/acceptability of the late-fallback regression.
   
   ### Was AI used for this PR?
   
   AI was used for investigation, implementation, differential review, 
regression tests, benchmark execution and this description. No human review is 
claimed.
   
   **PR code and description written by:**
   - [ ] Human
   - [x] AI
   
   **Reviewed before submission by:**
   - [ ] Human
   - [x] AI
   - [ ] Not reviewed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to