On Tuesday, 24 March 2020 18:31:29 UTC, Michael Jones wrote:
>
> You use them to stop the time charged against your benchmark.
>
> For example:
>
> bench:
> stop timer
> generate initial data
> start timer
> do test
>
Thanks Michael. What if the initial data has to be generated from scratch
on each iteration? Is the following expected to work?
bench:
b.N Loop
stop timer
generate new data
start timer
do test
>
> On Tue, Mar 24, 2020 at 10:47 AM Orson Cart <objectiv...@gmail.com
> <javascript:>> wrote:
>
>>
>> On Tuesday, 24 March 2020 16:49:55 UTC, Robert Engels wrote:
>>>
>>> Can you please succinctly explain the problem?
>>>
>>
>> Let's see. Benchmarks can yield incorrect results when b.StopTimer and
>> b.StartTimer are used.
>>
>> My reasoning:
>>
>> 1/ I have a benchmark that calls a single function. The reported duration
>> is T.
>> 2/ I have another benchmark which invokes the same function twice. I'd
>> expect the reported duration for this to be nominally 2T and within reason
>> it appears to be the case.
>> 3/ I have yet another benchmark which also invokes the same function
>> twice but it stops the benchmark timer before the first call and then
>> starts it after the the first call has completed. I'd expect the reported
>> duration to be nominally T again. It isn't. It's closer to 3T on my
>> hardware and I've seen much worse correlation on other hardware - almost 5T
>>
>> Now it's entirely possible that I'm misunderstanding something about how
>> b.StopTimer and b.StartTimer are intended to be used, hence my post here :)
>>
>>
>>
>>
>>
>>
>>
>>>
>>>
>>> On Mar 24, 2020, at 11:24 AM, Orson Cart <objectiv...@gmail.com> wrote:
>>>
>>>
>>> I posted this earlier but I realised that the code had a fundamental
>>> error in it. I've corrected here it but the underlying problem still exists.
>>>
>>> I've recently started using go test's benchmarks support and I'm
>>> particularly interested in understanding the benchmark timer functions.
>>> I've been getting results that I found surprising and I was wondering if
>>> anyone could explain what's going on here.
>>>
>>> The code below has three benchmarks that each invoke a single function
>>> (foo). The implementation of foo isn't important, it's just there to
>>> consume some time:
>>> - foo is called once per iteration in Benchmark1.
>>> - It's called twice per iteration in Benchmark2 so I'd expect
>>> Benchmark2's duration to be nominally twice that of Benchmark1.
>>> - It's also called twice per iteration in Benchmark3 but the first call
>>> is wrapped in b.StopTimer and b.startTimer calls. Because of this I'd have
>>> expected Benchmark3 to be about the same duration as Benchmark1
>>>
>>> Apologies for the length of the example but I didn't think it fair to
>>> ask the question and leave anything out.
>>>
>>> package demo_test
>>>
>>> import (
>>> "strconv"
>>> "testing"
>>> )
>>>
>>> var Foo1 []string
>>> var Foo2 []string
>>> var Count int = 32767
>>>
>>> func Benchmark1(b *testing.B) {
>>> for i := 0; i < b.N; i++{
>>> Foo1 = foo(Count)
>>> }
>>> }
>>>
>>> func Benchmark2(b *testing.B) {
>>> for i := 0; i < b.N; i++{
>>> Foo1 = foo(Count)
>>> Foo2 = foo(Count)
>>> }
>>> }
>>>
>>> func Benchmark3(b *testing.B) {
>>> for i := 0; i < b.N; i++{
>>> b.StopTimer()
>>> Foo1 = foo(Count)
>>> b.StartTimer()
>>> Foo2 = foo(Count)
>>> }
>>> }
>>>
>>> func foo(count int) []string{
>>> testData := []string{}
>>> for i:= 0; i < count; i++ {
>>> testData = append(testData, strconv.Itoa(i))
>>> }
>>>
>>> return testData
>>> }
>>>
>>>
>>> When the benchmarks are run the results are as follows:
>>>
>>> Benchmark1-4 351 3345215 ns/op
>>> Benchmark2-4 166 7206582 ns/op
>>> Benchmark3-4 334 3457907 ns/op
>>> PASS
>>> ok bar.com/benchmarks 6.881s
>>>
>>> OK benchmark3 is a little slower than Benchmark1 but that's not what's
>>> bothering me. It's this: if I now change Count to something much smaller
>>> the results are a surprise, at least to me. Here are the results when Count
>>> = 8:
>>>
>>> Benchmark1-4 2706196 442 ns/op
>>> Benchmark2-4 1357482 873 ns/op
>>> Benchmark3-4 840729 1387 ns/op
>>> PASS
>>> ok bar.com/benchmarks 23.547s
>>>
>>> The ratio of timings for Benchmark1 and Benchmark2 are roughly in line
>>> with expectations but I was surprised to see that the timings for
>>> Benchmark3 are now larger than those for Benchmark2.
>>>
>>> Can anyone explain this?
>>>
>>> TIA
>>> Orson
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golan...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/38f10940-a770-4fc5-86a9-19cc1371152a%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/golang-nuts/38f10940-a770-4fc5-86a9-19cc1371152a%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golan...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/1d291cef-420a-4f80-b719-e0654585c25d%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/golang-nuts/1d291cef-420a-4f80-b719-e0654585c25d%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
>
> *Michael T. jonesmichae...@gmail.com <javascript:>*
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/c3fcc6c5-9aa2-489c-adc4-c6b070ea6617%40googlegroups.com.
