mikamikasuki opened a new pull request, #11381: URL: https://github.com/apache/arrow-rs/pull/11381
# Which issue does this PR close? - Closes #11303 # Rationale for this change Loading a plain-encoded dictionary currently copies every value into an Arrow buffer, including entries that the data pages never reference. Keeping the encoded page until dictionary indices are read avoids materializing those unused entries. # What changes are included in this PR? The reader retains the dictionary page as shared `Bytes`, scans and stores value offsets, then copies and validates values as their indices are read. # Are these changes tested? Regression tests verify that the dictionary keeps the original page storage and that referenced invalid UTF-8 values still return an error. The Parquet library tests passed (1,351), and the Arrow reader integration tests passed (114; 1 ignored). Formatting, Clippy with warnings denied, and `git diff --check` also passed. No benchmark results are included. # Are there any significant user-facing changes? There is no public API change. For UTF-8 columns, invalid values that are never referenced by data pages are no longer validated; referenced values continue to be validated. Code assistance: AI assistance contributed to the implementation and regression tests; the contributor reviewed the changes and validation results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
