Diana Clarke created ARROW-11573:
------------------------------------
Summary: [Developer][Archery] Google benchmark now reports run type
Key: ARROW-11573
URL: https://issues.apache.org/jira/browse/ARROW-11573
Project: Apache Arrow
Issue Type: Bug
Components: Archery, Developer Tools
Reporter: Diana Clarke
Assignee: Diana Clarke
Google Benchmark now reports run type [1], so the following code and comment
can be updated.
{code}
Observations are found when running with `--benchmark_repetitions`. Sadly,
the format mixes values and aggregates, e.g.
RegressionSumKernel/32768/0 1 us 1 us 25.8077GB/s
RegressionSumKernel/32768/0 1 us 1 us 25.7066GB/s
RegressionSumKernel/32768/0 1 us 1 us 25.1481GB/s
RegressionSumKernel/32768/0 1 us 1 us 25.846GB/s
RegressionSumKernel/32768/0 1 us 1 us 25.6453GB/s
RegressionSumKernel/32768/0_mean 1 us 1 us 25.6307GB/s
RegressionSumKernel/32768/0_median 1 us 1 us 25.7066GB/s
RegressionSumKernel/32768/0_stddev 0 us 0 us 288.046MB/s
As from benchmark v1.4.1 (2019-04-24), the only way to differentiate an
actual run from the aggregates, is to match on the benchmark name. The
aggregates will be appended with `_$agg_name`.
This class encapsulate the logic to separate runs from aggregate . This is
hopefully avoided in benchmark's master version with a separate json
attribute.
{code}
{code}
@property
def is_agg(self):
""" Indicate if the observation is a run or an aggregate. """
suffixes = ["_mean", "_median", "_stddev"]
return any(map(lambda x: self._name.endswith(x), suffixes))
{code}
Here's example output (note the aggregate vs the actual observation):
{code}
{'aggregate_name': 'mean',
'cpu_time': 9818703.124999983,
'items_per_second': 26700744.55186333,
'iterations': 3,
'name': 'TakeStringRandomIndicesWithNulls/262144/0_mean',
'null_percent': 0.0,
'real_time': 10138621.349445505,
'repetitions': 0,
'run_name': 'TakeStringRandomIndicesWithNulls/262144/0',
'run_type': 'aggregate',
'size': 262144.0,
'threads': 1,
'time_unit': 'ns'},
{'cpu_time': 9718937.499999996,
'items_per_second': 26972495.707478322,
'iterations': 64,
'name': 'TakeStringRandomIndicesWithNulls/262144/0',
'null_percent': 0.0,
'real_time': 10297947.859726265,
'repetition_index': 2,
'repetitions': 0,
'run_name': 'TakeStringRandomIndicesWithNulls/262144/0',
'run_type': 'iteration',
'size': 262144.0,
'threads': 1,
'time_unit': 'ns'},
{code}
[1]
https://github.com/google/benchmark/commit/8688c5c4cfa1527ceca2136b2a738d9712a01890
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
