cetra3 opened a new pull request, #12310: URL: https://github.com/apache/arrow-rs/pull/12310
# Which issue does this PR close? - Part of #11511 This PR is stacked on #12308 and #12309. Its own change is the third commit, "perf(variant): read unshredded columns through views in `variant_get`". It is a draft until they merge. # Rationale for this change For a path over unshredded data, `variant_get` builds a `Variant` for every row and then walks the path through it. Building the `Variant` checks the first and last offsets of the metadata and of each object and list, and each field lookup validates the UTF-8 of the names that it probes. Rows that do not contain the path do all of that and produce a null. # What changes are included in this PR? - `VariantView::get_path` follows a path of field names and list indexes through the views of #12309. - `variant_get` reads a column that has no `typed_value` through it, for field and index paths and for typed and `Variant` output. A row reads only the headers and names along the path, and decodes only the value that it finds. - A crate-private `BinaryColumn` resolves the array type of the `metadata` and `value` columns once, not on every row. The gain comes from skipping the per-row `Variant` construction and the UTF-8 validation of probed names. # Are these changes tested? The existing `variant_get` tests cover the new path, because their unshredded input now takes it. `VariantView::get_path` has a test that compares it with `Variant::get_path`. Benchmarks (local run, this PR against #12309, x86_64 Linux): | benchmark | #12309 | this PR | change | |---|---:|---:|---:| | `variant_get_binary_from_string` | 316.95 µs | 207.79 µs | −34% | | `variant_get_binary_from_unshredded_binary` | 264.24 µs | 194.71 µs | −26% | | `variant_get_list_index_all_oob_int64_64_rows` | 421.86 ns | 418.12 ns | −1% | | `variant_get_list_index_shredded` | 26.80 µs | 25.72 µs | −4% | | `variant_get_list_index_unshredded` | 576.96 µs | 308.11 µs | −47% | | `variant_get_shredded_utf8` | 342.91 ns | 341.04 ns | −1% | | `variant_get_unshredded_object_path_262k_rows` | 22.79 ms | 14.98 ms | −34% | | `variant_get_utf8_from_date` | 386.90 µs | 306.39 µs | −21% | | `variant_get_utf8_from_decimal` | 265.38 µs | 186.75 µs | −30% | | `variant_get_utf8_from_float` | 255.41 µs | 176.07 µs | −31% | | `variant_get_utf8_from_int` | 256.21 µs | 179.43 µs | −30% | | `variant_get_utf8_from_list` | 369.64 µs | 323.23 µs | −13% | | `variant_get_utf8_from_map_in_list` | 367.95 µs | 328.31 µs | −11% | | `variant_get_utf8_from_time` | 452.92 µs | 370.30 µs | −18% | | `variant_get_utf8_from_timestamp_with_timezone` | 320.17 µs | 241.92 µs | −24% | | `variant_get_utf8_from_timestamp_without_timezone` | 318.81 µs | 243.21 µs | −24% | | `variant_get_utf8_from_unshredded_string` | 321.26 µs | 214.52 µs | −33% | No benchmark in this set is slower. The shredded benchmarks (`variant_get_shredded_utf8`, `variant_get_list_index_shredded`, `variant_get_list_index_all_oob_int64_64_rows`) do not take the new path. # Are there any user-facing changes? New public API in `parquet-variant`: `VariantView::get_path`. On malformed variant bytes, `variant_get` on an unshredded column returns an error, or a null under safe casting. Before, this path could panic. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
