Rich-T-kid opened a new pull request, #12126: URL: https://github.com/apache/arrow-rs/pull/12126
# Which issue does this PR close? - Closes #11748 - Part of apache/datafusion#24111 # Rationale for this change DataFusion's `binary_as_string` and RLE dictionary options need an unannotated Parquet `BYTE_ARRAY` to map directly to `Dictionary<K, Utf8>`. The reader currently chooses the dictionary value type from the Parquet annotation, so it still builds a `Binary` child when the supplied Arrow schema requests `Utf8`. DataFusion then has to expand and rebuild the dictionary row by row. # What changes are included in this PR? Decode dictionary pages using their physical Parquet value type, convert only the distinct dictionary values to the value type requested by the supplied Arrow schema, and reuse the converted values across batches. This produces `Dictionary<K, Utf8>` directly while preserving the encoded keys. Plain-encoded pages continue to use the existing dictionary-building fallback. The same path handles compatible string/binary and view-type requests, including UTF-8 validation when a binary dictionary is requested as a string dictionary. # Are these changes tested? Yes. Regression tests cover: - unannotated binary dictionaries requested as `Dictionary<Int32, Utf8>` across multiple batches - preserved keys and shared dictionary values - invalid UTF-8 input - annotated UTF-8 dictionaries requested as binary The Parquet crate's byte-array dictionary and supplied-schema test modules pass, as does `cargo clippy -p parquet --all-targets --all-features -- -D warnings`. ## Benchmarks The Arrow microbenchmark uses 5 million rows, 20,000 distinct 24-byte values, no nulls, and dictionary-encoded unannotated Parquet `BYTE_ARRAY` data: | Reader output | Median | |---|---:| | `Dictionary(Int32, Binary)` | 2.48 ms | | `Dictionary(Int32, Utf8)` (this PR) | 2.61 ms | | `Utf8View` | 4.87 ms | | `Utf8View` then cast to dictionary | 126.41 ms | For an end-to-end check, I built DataFusion main (`32dd1d2c`) twice against Arrow main (`62a3c43f`) and this PR (`ea86d977`), with RLE-to-dictionary enabled on both sides. The candidate build requests `Dictionary(Int32, Utf8)` directly for binary-as-string columns. ClickBench used the 100-file, 14.7 GB dataset; each result below is the median of 10 iterations in balanced baseline/candidate order: | Query | Baseline | This PR | Change | |---|---:|---:|---:| | Q10 | 152.39 ms | 49.75 ms | 3.06x faster | | Q11 | 153.36 ms | 59.12 ms | 2.59x faster | | Q22 | 2,755.64 ms | 1,918.13 ms | 1.44x faster | | Q23 | 8,704.25 ms | 5,001.62 ms | 1.74x faster | | Q25 | 362.55 ms | 229.04 ms | 1.58x faster | | Q37 | 69.80 ms | 41.44 ms | 1.68x faster | | Q32 | 805.93 ms | 840.29 ms | 4.3% slower | Five-iteration SF1 runs were also directionally positive: the sum of per-query medians decreased 4.3% for TPC-H and 3.1% for TPC-DS. These smaller suites are included as broad regression checks; ClickBench is the representative binary-as-string workload. # Are there any significant user-facing changes? Supplied dictionary schemas now produce columns with the requested compatible string or binary value type. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
