adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733724577
## Benchmark summary
`variant_get` on an unshredded object path is **2.89x faster**: 16.0ms
against 45.8ms for 262k rows.
Across 39 benchmarks in three suites: one gets much faster, three get about
2% slower, and 35 are flat.
| bench | change runs | control runs | verdict |
| --- | --- | --- | --- |
| `variant_get_unshredded_object_path_262k_rows` | 0.35 / 0.35 / 0.34 | 1.01
/ 1.00 / 1.00 | **2.89x faster** |
| `shred_variant_unmatched_object_8k_rows` | 1.02 / 1.02 / 1.02 | 1.00 /
1.00 / 1.00 | 2% slower |
| `shred_variant_partial_object_8k_rows` | 1.01 / 1.02 / 1.02 | 1.00 / 1.00
/ 1.00 | 2% slower |
| `batch_json_string_to_variant json_list 8k string` | 1.04 / 1.04 / 1.05 |
1.02 / 1.02 / 1.02 | 2% slower |
| the other 35 | - | - | flat |
Each number is the ratio of this branch to its merge base. Every job has a
paired main-vs-main control job, because several of these benchmarks move by 1
to 5% on identical code. Twelve change jobs and twelve control jobs in total.
The PR title says 79%. That number is from the original measurement in July.
The correct number for this branch on this runner is 2.89x, which is a 65%
decrease in latency.
### About the three benchmarks that get slower
None of the three can reach the code this PR changes, and the direct
microbenchmarks of the changed functions are flat:
- `object_field_name` is a tight loop on `VariantObject::field_name`, the
function this PR splits into `try_field_name` plus `try_field_id`. It is flat:
1.00 and 1.01 against a control at 1.00.
- `object_get_by_name` and `object_iter` are also flat.
- `json_to_variant` and `shred_variant` write variants. They do not call the
new path resolution at all.
All three benchmarks that move are in `parquet-variant-compute`, which this
PR grows by about 180 lines. All 22 benchmarks in `parquet-variant`, which
grows by about 30 lines of non-test code, are flat. That points at code
placement rather than a cost in the new logic.
Two things I ruled out:
- **Enum growth.** `VariantPathRowBuilder` grows from 32 to 88 bytes, and it
is one variant of the `VariantToArrowRowBuilder` enum that `shred_variant`
uses. That enum is 224 bytes both on main and here, because a different variant
is the largest one.
- **An `#[inline]` on `try_field_id`.** It changes the emitted IR, but the
benchmark that measures the split is already flat, so there was nothing to
recover. I dropped it.
These benchmarks are very sensitive to placement. On my laptop,
`variant_get_shredded_utf8` reported +7.8% for this branch, and then +14.0%
after I moved the new functions to a different place in the same file and
changed nothing else. On this runner it is flat.
My suggestion is to take about 2% on three benchmarks this PR does not
target, for 2.89x on the one it does. I am happy to look further if you would
rather not.
<details><summary>The 35 flat benchmarks, and how these runs were
made</summary>
- `variant_kernels`: `variant_get_primitive`, `variant_get_shredded_utf8`,
`variant_array_builder_build_262k_small_values`, and four more
`batch_json_string_to_variant` cases (`repeated_struct 8k string`,
`random_json(2633 bytes per document)`, `object - 1 depth(100 fields)`, `object
- 1 depth(200 fields)`)
- `variant_validation`: `bench_validate_large_object`,
`bench_validate_complex_object`, `bench_validate_large_nested_list`,
`metadata_iter/{8,32,128}`, `object_field_name/{8,32,128}`,
`object_iter/{8,32,128}`, `object_get_by_name/{8,32,128}`
- `variant_builder`: `bench_object_same_schema`,
`bench_object_field_names_reverse_order`, `bench_object_unknown_schema`,
`bench_object_partially_same_schema`, `bench_object_list_same_schema`,
`bench_object_list_unknown_schema`, `bench_object_list_partially_same_schema`,
`bench_extend_metadata_builder`,
`validation/{validated_construction,unvalidated_construction,validation_cost}`,
`iteration/{validated_iteration,unvalidated_fallible_iteration}`
Two of those need a note, because the ratio on its own is misleading:
- `bench_object_same_schema` reads a steady 1.05 in all three change runs,
with error bars of 0.2%. Its three control runs read 1.03, 1.04 and 1.05 on
identical code. The runner benchmarks the baseline side first on the same pod,
and this benchmark is 3 to 5% slower on the second side whatever is in it. It
is flat.
- One `variant_validation` job reported `object_field_name` at 1.07 to 1.08,
but with 17% error bars on one side only (79.6+/-13.67ns against
73.9+/-2.02ns). Two other runs of the same commit are flat with 3% error bars.
That pod was disturbed, so I discarded the run.
Run configuration, on the `adriangbot` GKE runner (`c4a-highmem-16`, 12
vCPU, Neoverse-V2), three change jobs and three control jobs of each:
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
```yaml
run benchmark variant_validation variant_builder
```
A control job pins both sides to the merge base,
`1a6630bc2c198955a9d24b34ca560dcbb9167a7f`:
```yaml
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
```
A control job compares two identical binaries. It gives the noise floor and
it shows the tilt above, but it cannot separate a code placement effect from a
real cost, because placement only differs between different binaries.
`parquet-variant` and `parquet-variant-compute` hold every variant benchmark
in the repository. The `parquet` crate has none, so there is nothing else to
measure.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
