setup upfront shouldn't be a problem b.N is constant throughout the function lifetime, the entire function is called multiple times during benchmarking
On Tuesday, March 24, 2020 at 9:56:19 PM UTC+1, Orson Cart wrote: > > On Tuesday, 24 March 2020 20:47:07 UTC, Robert Engels wrote: >> >> One way to handle this is to generate all of the data up front in an >> array and then just index into the array based on the run. >> > > Yeah, I had thought of that before posting but then I'd have to decide on > a value for b.N. I was trying to roll with the idea of the framework > establishing a value. To be honest if it wasn't for the fact that I was > just curious as to why it wasn't working I'd just have settled for an array > with a fixed b.N. > > All part of my golang learning experience > > > >> >> On Mar 24, 2020, at 3:42 PM, Orson Cart <objectiv...@gmail.com> wrote: >> >> >> >> >> On Tuesday, 24 March 2020 20:27:39 UTC, Adrian Ratnapala wrote: >>> >>> ... >> >> So that sounds like the use-case is to call Stop-, StartTimer once >>> before you enter the main loop of the benchmark. They not efficient >>> enough for a tight inner loop. >>> >> >> Thanks for the input and I think that you are probably correct. It would >> be nice if the docs said that though. >> >> The code that I'm profiling wasn't written by me. It's fast but it >> modifies its input data so it needs new data on each iteration. The >> implementation is non-trivial and it might be a maintenance headache. I've >> been asked to benchmark it along with less complicated implementations. It >> was whilst trying to set up per-iteration test data that I came across this >> issue. >> >> >> >> >> >> >> >> >> >>> >>> On Wed, 25 Mar 2020 at 06:51, Jake Montgomery <jake...@gmail.com> >>> wrote: >>> > >>> > Strange. I hope someone has a real answer for you. >>> > >>> > In the meantime, you can simplify your example to demonstrate the >>> issue: >>> > >>> > package demo_test >>> > >>> > import ( >>> > "testing" >>> > ) >>> > >>> > var Foo1 []string >>> > var Count int = 8 >>> > >>> > func Benchmark1(b *testing.B) { >>> > for i := 0; i < b.N; i++ { >>> > Foo1 = foo(Count) >>> > } >>> > } >>> > >>> > func Benchmark2(b *testing.B) { >>> > b.StopTimer() >>> > for i := 0; i < b.N; i++ { >>> > // Hypothetical setup here >>> > b.StartTimer() >>> > Foo1 = foo(Count) >>> > b.StopTimer() >>> > } >>> > } >>> > >>> > >>> > func foo(count int) []string { >>> > testData := []string{} >>> > for i := 0; i < count; i++ { >>> > testData = append(testData, "a") >>> > } >>> > >>> > return testData >>> > } >>> > >>> > I get: >>> > >>> > goos: windows >>> > goarch: amd64 >>> > Benchmark1-4 2101567 584 ns/op >>> > Benchmark2-4 1000000 1668 ns/op >>> > PASS >>> > >>> > So it appears that StopTimer() and StartTimer() are introducing some >>> overhead. I am surprised that it is this large. >>> > >>> > Good Luck >>> > >>> > >>> > >>> > On Tuesday, March 24, 2020 at 12:24:08 PM UTC-4, Orson Cart wrote: >>> >> >>> >> I posted this earlier but I realised that the code had a fundamental >>> error in it. I've corrected here it but the underlying problem still >>> exists. >>> >> >>> >> I've recently started using go test's benchmarks support and I'm >>> particularly interested in understanding the benchmark timer functions. >>> I've been getting results that I found surprising and I was wondering if >>> anyone could explain what's going on here. >>> >> >>> >> The code below has three benchmarks that each invoke a single >>> function (foo). The implementation of foo isn't important, it's just there >>> to consume some time: >>> >> - foo is called once per iteration in Benchmark1. >>> >> - It's called twice per iteration in Benchmark2 so I'd expect >>> Benchmark2's duration to be nominally twice that of Benchmark1. >>> >> - It's also called twice per iteration in Benchmark3 but the first >>> call is wrapped in b.StopTimer and b.startTimer calls. Because of this I'd >>> have expected Benchmark3 to be about the same duration as Benchmark1 >>> >> >>> >> Apologies for the length of the example but I didn't think it fair to >>> ask the question and leave anything out. >>> >> >>> >> package demo_test >>> >> >>> >> import ( >>> >> "strconv" >>> >> "testing" >>> >> ) >>> >> >>> >> var Foo1 []string >>> >> var Foo2 []string >>> >> var Count int = 32767 >>> >> >>> >> func Benchmark1(b *testing.B) { >>> >> for i := 0; i < b.N; i++{ >>> >> Foo1 = foo(Count) >>> >> } >>> >> } >>> >> >>> >> func Benchmark2(b *testing.B) { >>> >> for i := 0; i < b.N; i++{ >>> >> Foo1 = foo(Count) >>> >> Foo2 = foo(Count) >>> >> } >>> >> } >>> >> >>> >> func Benchmark3(b *testing.B) { >>> >> for i := 0; i < b.N; i++{ >>> >> b.StopTimer() >>> >> Foo1 = foo(Count) >>> >> b.StartTimer() >>> >> Foo2 = foo(Count) >>> >> } >>> >> } >>> >> >>> >> func foo(count int) []string{ >>> >> testData := []string{} >>> >> for i:= 0; i < count; i++ { >>> >> testData = append(testData, strconv.Itoa(i)) >>> >> } >>> >> >>> >> return testData >>> >> } >>> >> >>> >> >>> >> When the benchmarks are run the results are as follows: >>> >> >>> >> Benchmark1-4 351 3345215 ns/op >>> >> Benchmark2-4 166 7206582 ns/op >>> >> Benchmark3-4 334 3457907 ns/op >>> >> PASS >>> >> ok bar.com/benchmarks 6.881s >>> >> >>> >> OK benchmark3 is a little slower than Benchmark1 but that's not >>> what's bothering me. It's this: if I now change Count to something much >>> smaller the results are a surprise, at least to me. Here are the results >>> when Count = 8: >>> >> >>> >> Benchmark1-4 2706196 442 ns/op >>> >> Benchmark2-4 1357482 873 ns/op >>> >> Benchmark3-4 840729 1387 ns/op >>> >> PASS >>> >> ok bar.com/benchmarks 23.547s >>> >> >>> >> The ratio of timings for Benchmark1 and Benchmark2 are roughly in >>> line with expectations but I was surprised to see that the timings for >>> Benchmark3 are now larger than those for Benchmark2. >>> >> >>> >> Can anyone explain this? >>> >> >>> >> TIA >>> >> Orson >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to golan...@googlegroups.com. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/feda7e38-5d1f-43cf-b0cd-98db0a94d3c9%40googlegroups.com. >>> >>> >>> >>> >>> >>> -- >>> Adrian Ratnapala >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golan...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/291af5de-8a01-4c30-98bb-d4765e3a5333%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/291af5de-8a01-4c30-98bb-d4765e3a5333%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e690b837-75e6-4c6e-ba8a-8cc00353a06e%40googlegroups.com.