cetra3 opened a new pull request, #12311:
URL: https://github.com/apache/arrow-rs/pull/12311

   # Which issue does this PR close?
   
   - Part of #11511
   
   This PR is stacked on #12308, #12309 and #12310. Its own change is the 
fourth commit, "perf(variant): copy leftover object fields by field id when 
shredding". It is a draft until they merge.
   
   # Rationale for this change
   
   When `shred_variant` and `unshred_variant` copy an object field that is not 
shredded, they decode it into a `Variant`, look its name up in the metadata 
dictionary to get the field id back, and encode the value again. The output 
uses the same metadata dictionary as the input, so the field id and the encoded 
bytes are already correct.
   
   # What changes are included in this PR?
   
   - `ObjectBuilder::insert_encoded` adds a field by its field id and its 
encoded bytes.
   - `VariantView::bytes` returns the exact encoding of a value.
   - `shred_variant` and `unshred_variant` copy leftover object fields through 
them. They match field names as bytes against a sorted list of the shredded 
field names.
   
   # Are these changes tested?
   
   The existing shredding tests cover the changed code. `VariantView::bytes` 
has tests for every primitive type, and for the fields of an object and the 
elements of a list, whose encodings must tile the value region of their parent.
   
   Benchmarks (local run, this PR against #12310, x86_64 Linux):
   
   | benchmark | #12310 | this PR | change |
   |---|---:|---:|---:|
   | `shred_variant_partial_object_8k_rows` | 10.90 ms | 9.42 ms | −14% |
   | `shred_variant_unmatched_object_8k_rows` | 14.43 ms | 11.36 ms | −21% |
   | `unshred_variant_list` | 5.44 ms | 5.59 ms | +3% |
   | `unshred_variant_list_of_objects` | 11.26 ms | 11.46 ms | +2% |
   | `unshred_variant_nested_fallback` | 74.62 ms | 75.06 ms | +1% |
   | `unshred_variant_object` | 2.93 ms | 3.05 ms | +4% |
   | `unshred_variant_partial_object` | 24.13 ms | 21.81 ms | −10% |
   | `unshred_variant_partial_object_unsorted_metadata` | 107.97 ms | 62.79 ms 
| −42% |
   | `unshred_variant_primitive` | 263.33 µs | 260.74 µs | −1% |
   | `variant_get_unshredded_object_path_262k_rows` | 14.98 ms | 13.61 ms | −9% 
|
   | `variant_get_list_index_unshredded` | 308.11 µs | 283.84 µs | −8% |
   | `variant_get_utf8_from_unshredded_string` | 214.52 µs | 233.12 µs | +9% |
   | `variant_get_binary_from_string` | 207.79 µs | 226.29 µs | +9% |
   
   Slower inputs: `unshred_variant_object` is 4% slower and 
`unshred_variant_list` is 3% slower. `variant_get_utf8_from_unshredded_string` 
and `variant_get_binary_from_string` are 9% slower in this run, although this 
PR does not change `variant_get`; these two benchmarks have moved by 10–15% 
with unrelated changes on this machine.
   
   # Are there any user-facing changes?
   
   New public API in `parquet-variant`: `ObjectBuilder::insert_encoded` and 
`VariantView::bytes`.
   
   On malformed variant bytes, `shred_variant` and the leftover copy in 
`unshred_variant` return an error. Before, these paths could panic.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to