sdf-jkl opened a new issue, #11101: URL: https://github.com/apache/arrow-rs/issues/11101
### Is your feature request related to a problem or challenge? Variant changes need benchmarks that exercise the affected execution paths and make their allocation costs visible. Existing coverage includes binary builders, validation and iteration, JSON ingestion, builder finalization, object shredding, and a few `variant_get` cases. Important gaps remain in list indexing, nested shredded traversal, typed conversion, reconstruction, and allocation measurements for these operations. ### Describe the solution you'd like The goal is a reproducible baseline for reviewing runtime, allocation, and memory changes across the Variant APIs. Organize coverage by materially different execution paths, using a manageable set of representative workloads. #### Existing coverage Audited the registered benchmarks and their timed bodies on upstream main at [c60ac3fc66](https://github.com/apache/arrow-rs/commit/c60ac3fc66b0a0d105cfd4d1280b9753f426079f) (2026-09-15), including searches for Variant use in other crates' benchmarks. The following coverage is already present and is excluded from the work to add. Reuse these cases as controls and as inputs for new memory measurements. | Existing coverage | Benchmark names / groups | | --- | --- | | Untyped primitive extraction and perfectly shredded UTF-8 extraction, both with an empty path; unshredded object-field extraction as Int32 | `variant_get_primitive`, `variant_get_shredded_utf8`, `variant_get_unshredded_object_path_262k_rows` in [variant_kernels][kernels] | | Partial and unmatched object shredding, including leftover binary values | `shred_variant_partial_object_8k_rows`, `shred_variant_unmatched_object_8k_rows` in [variant_kernels][kernels] | | JSON ingestion of repeated objects, variable-length numeric lists including empty lists, and random mixed/nested JSON; appending pre-parsed wide objects with 100/200 fields | `batch_json_string_to_variant ...` in [variant_kernels][kernels]. The wide-object cases time `append_json` and array construction, with JSON parsing outside timing. | | Array-builder finalization for 262,144 small values | `variant_array_builder_build_262k_small_values` in [variant_kernels][kernels], delivered by [#10621](https://github.com/apache/arrow-rs/issues/10621) / [#10640](https://github.com/apache/arrow-rs/pull/10640) | | Objects and lists of objects with shared, partially shared, and random schemas; nested lists/objects and mixed string lengths; reverse-order field insertion; metadata-builder extension | `bench_object_*`, `bench_extend_metadata_builder` in [variant_builder][builder] | | Validated/unvalidated construction, explicit full validation, and validated/fallible list iteration | `validation/*`, `iteration/*` in [variant_builder][builder] | | Validation of large/complex objects and large nested lists; metadata iteration, indexed object field-name access, object iteration, and lookup by name at 8/32/128 fields | `bench_validate_*`, `metadata_iter/*`, `object_field_name/*`, `object_iter/*`, `object_get_by_name/*` in [variant_validation][validation] | #### Work items - [ ] **Missing `variant_get` traversal paths.** Add direct list indexing, shredded object-field traversal, list-of-list, list-of-object, and object-of-list paths. Compare unshredded, fully shredded, and partially shredded inputs where applicable. Include Variant output and a matching typed output; exercise both direct column reuse and binary fallback. Use shallow and deeper paths, and small and larger batches, to expose fixed and per-step overhead. Reuse the existing unshredded object-path and empty-path cases as controls. - List cases: in-bounds, mixed empty/nonempty lists, all out-of-bounds, parent Arrow nulls, and explicit Variant-null elements. - Include sliced arrays and representative list/list-view layouts, adding variants where their implementation differs. - Keep struct-only traversal as a control for changes to shared traversal state. - [ ] **Typed unshredded conversions in `variant_get`.** Track [#10988](https://github.com/apache/arrow-rs/issues/10988): direct extraction, conversion/formatting, and nested values, initially targeting UTF-8 and Binary destinations. Add representative numeric conversions and safe-cast failures that differ from the existing Int32 object-field extraction. - [ ] **Missing shredding cases.** Extend `shred_variant` coverage with fully matching schemas and nested list/object shredding. Retain the existing partial/unmatched flat-object benchmarks as controls. - [ ] **Unshredding.** Add dedicated `unshred_variant` benchmarks for the unshredded fast path, fully and partially shredded reconstruction, nested lists/objects, and binary fallback. No registered benchmark currently calls this kernel directly. - [ ] **JSON output.** Track `variant_to_json` coverage in [#10906](https://github.com/apache/arrow-rs/issues/10906): scalars, small/wide objects, shredded input, and encoded metadata. Existing JSON-ingestion benchmarks remain the controls for the other direction. - [ ] **Arrow-to-Variant conversion.** Add `cast_to_variant` benchmarks for primitive/string arrays and nested lists/structs, including representative nulls and layouts where buffer reuse differs. Existing builder and JSON-ingestion benchmarks do not invoke this API. - [ ] **Variant allocation and memory measurements.** Add repeatable measurements for existing and new workloads, starting with path traversal and shredding. Reuse or adapt the tracking allocator and Criterion measurement approach in [arrow_reader_peak_memory][memory], which currently benchmarks Parquet array readers. Report allocations and allocated bytes per call, peak additional live heap bytes, and memory retained by the result. Keep shared input buffers distinct from newly allocated output; account for unique backing allocations when reporting retained memory. Measure inline sizes of relevant state types separately when their representation changes, with the target architecture recorded. - [ ] **Reproduction and reporting.** Document commands, benchmark filters, workload definitions, and how to compare an identical benchmark on the base and proposed implementation. Make the focused suites runnable through the existing benchmark workflow. #### Measurement requirements - Use deterministic inputs and stable benchmark names. Document batch size, nesting, value sizes, null distribution, and shredding schema. - Generate inputs, parse paths, and prepare schemas outside the timed operation. For consuming builders, use batched setup so a finalization benchmark isolates finalization. State whether output destruction is timed. - Validate expected values and Arrow validity outside timing. Missing results and explicit Variant nulls must remain distinct. - Report iteration time and rows/second; include bytes/second where the byte count has a clear definition. - Measure allocations separately from timing so instrumentation does not distort the runtime comparison. Exclude fixture and harness setup. Process-wide peak RSS alone is insufficient to explain per-call memory costs. - Use identical inputs, compiler settings, and hardware for comparisons; record commit IDs and uncertainty. Preserve raw results so small differences can be assessed against noise. - Add focused benchmarks in benchmark-only PRs where practical, then use them to evaluate implementation changes. Explain added ownership or allocation and measure the tradeoff; borrowing and Arc sharing are useful mechanisms, not substitutes for measurements. [kernels]: https://github.com/apache/arrow-rs/blob/c60ac3fc66b0a0d105cfd4d1280b9753f426079f/parquet-variant-compute/benches/variant_kernels.rs [builder]: https://github.com/apache/arrow-rs/blob/c60ac3fc66b0a0d105cfd4d1280b9753f426079f/parquet-variant/benches/variant_builder.rs [validation]: https://github.com/apache/arrow-rs/blob/c60ac3fc66b0a0d105cfd4d1280b9753f426079f/parquet-variant/benches/variant_validation.rs [memory]: https://github.com/apache/arrow-rs/blob/c60ac3fc66b0a0d105cfd4d1280b9753f426079f/parquet/benches/arrow_reader_peak_memory.rs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
