cetra3 opened a new pull request, #12310:
URL: https://github.com/apache/arrow-rs/pull/12310

   # Which issue does this PR close?
   
   - Part of #11511
   
   This PR is stacked on #12308 and #12309. Its own change is the third commit, 
"perf(variant): read unshredded columns through views in `variant_get`". It is 
a draft until they merge.
   
   # Rationale for this change
   
   For a path over unshredded data, `variant_get` builds a `Variant` for every 
row and then walks the path through it. Building the `Variant` checks the first 
and last offsets of the metadata and of each object and list, and each field 
lookup validates the UTF-8 of the names that it probes. Rows that do not 
contain the path do all of that and produce a null.
   
   # What changes are included in this PR?
   
   - `VariantView::get_path` follows a path of field names and list indexes 
through the views of #12309.
   - `variant_get` reads a column that has no `typed_value` through it, for 
field and index paths and for typed and `Variant` output. A row reads only the 
headers and names along the path, and decodes only the value that it finds.
   - A crate-private `BinaryColumn` resolves the array type of the `metadata` 
and `value` columns once, not on every row.
   
   The gain comes from skipping the per-row `Variant` construction and the 
UTF-8 validation of probed names.
   
   # Are these changes tested?
   
   The existing `variant_get` tests cover the new path, because their 
unshredded input now takes it. `VariantView::get_path` has a test that compares 
it with `Variant::get_path`.
   
   Benchmarks (local run, this PR against #12309, x86_64 Linux):
   
   | benchmark | #12309 | this PR | change |
   |---|---:|---:|---:|
   | `variant_get_binary_from_string` | 316.95 µs | 207.79 µs | −34% |
   | `variant_get_binary_from_unshredded_binary` | 264.24 µs | 194.71 µs | −26% 
|
   | `variant_get_list_index_all_oob_int64_64_rows` | 421.86 ns | 418.12 ns | 
−1% |
   | `variant_get_list_index_shredded` | 26.80 µs | 25.72 µs | −4% |
   | `variant_get_list_index_unshredded` | 576.96 µs | 308.11 µs | −47% |
   | `variant_get_shredded_utf8` | 342.91 ns | 341.04 ns | −1% |
   | `variant_get_unshredded_object_path_262k_rows` | 22.79 ms | 14.98 ms | 
−34% |
   | `variant_get_utf8_from_date` | 386.90 µs | 306.39 µs | −21% |
   | `variant_get_utf8_from_decimal` | 265.38 µs | 186.75 µs | −30% |
   | `variant_get_utf8_from_float` | 255.41 µs | 176.07 µs | −31% |
   | `variant_get_utf8_from_int` | 256.21 µs | 179.43 µs | −30% |
   | `variant_get_utf8_from_list` | 369.64 µs | 323.23 µs | −13% |
   | `variant_get_utf8_from_map_in_list` | 367.95 µs | 328.31 µs | −11% |
   | `variant_get_utf8_from_time` | 452.92 µs | 370.30 µs | −18% |
   | `variant_get_utf8_from_timestamp_with_timezone` | 320.17 µs | 241.92 µs | 
−24% |
   | `variant_get_utf8_from_timestamp_without_timezone` | 318.81 µs | 243.21 µs 
| −24% |
   | `variant_get_utf8_from_unshredded_string` | 321.26 µs | 214.52 µs | −33% |
   
   No benchmark in this set is slower. The shredded benchmarks 
(`variant_get_shredded_utf8`, `variant_get_list_index_shredded`, 
`variant_get_list_index_all_oob_int64_64_rows`) do not take the new path.
   
   # Are there any user-facing changes?
   
   New public API in `parquet-variant`: `VariantView::get_path`.
   
   On malformed variant bytes, `variant_get` on an unshredded column returns an 
error, or a null under safe casting. Before, this path could panic.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to