Wes McKinney created ARROW-6098: ----------------------------------- Summary: [C++] Partially mitigating CPU scaling effects in benchmarks Key: ARROW-6098 URL: https://issues.apache.org/jira/browse/ARROW-6098 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Wes McKinney
We have a lot of benchmarks that return results based on a single iteration {code} (arrow-3.7) 10:46 ~/code/arrow/cpp/build (master)$ ./release/arrow-builder-benchmark --benchmark_filter=Dict 2019-08-01 10:46:03 Running ./release/arrow-builder-benchmark Run on (12 X 4400 MHz CPU s) CPU Caches: L1 Data 32K (x6) L1 Instruction 32K (x6) L2 Unified 256K (x6) L3 Unified 12288K (x1) ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. --------------------------------------------------------------------------- Benchmark Time CPU Iterations --------------------------------------------------------------------------- BuildInt64DictionaryArrayRandom 622889286 ns 622864485 ns 1 411.004MB/s BuildInt64DictionaryArraySequential 546764048 ns 545992395 ns 1 468.871MB/s BuildInt64DictionaryArraySimilar 737759293 ns 737696850 ns 1 347.026MB/s BuildStringDictionaryArray 985433473 ns 985363901 ns 1 346.608MB/s (arrow-3.7) 10:46 ~/code/arrow/cpp/build (master)$ ./release/arrow-builder-benchmark --benchmark_filter=Dict 2019-08-01 10:46:09 Running ./release/arrow-builder-benchmark Run on (12 X 4400 MHz CPU s) CPU Caches: L1 Data 32K (x6) L1 Instruction 32K (x6) L2 Unified 256K (x6) L3 Unified 12288K (x1) ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. --------------------------------------------------------------------------- Benchmark Time CPU Iterations --------------------------------------------------------------------------- BuildInt64DictionaryArrayRandom 527063570 ns 527044023 ns 1 485.728MB/s BuildInt64DictionaryArraySequential 566285427 ns 566270336 ns 1 452.081MB/s BuildInt64DictionaryArraySimilar 762954193 ns 762332297 ns 1 335.812MB/s BuildStringDictionaryArray 991095766 ns 991018875 ns 1 344.63MB/s {code} I'm sure the result here is being heavily affected by CPU scaling but I think we can mitigate the impact of CPU scaling by using the `MinTime`. I find that adding `MinTime(1.0)` to these particular benchmarks makes them more consistent -- This message was sent by Atlassian JIRA (v7.6.14#76016)