You didn't show us the actual commands you are running to produce the two log files. It is also unclear why you expect benchstat to report a difference. In other words: insufficient information for anyone to make even a guess why benchstat isn't producing the output you expect.
On Tue, Jul 27, 2021 at 8:56 PM Tong Sun <suntong...@gmail.com> wrote: > Hi, > > I'm trying to understand the benchstat's output. > The benchstat that I have is fresh from golang.org/x/perf/cmd/, installed > hours ago. > > This is understandable: > > name old time/op new time/op delta > FormatEmoji-2 1.79µs ± 0% 1.82µs ± 1% ~ (p=0.100 n=3+3) > FormatEmoji-4 1.99µs ± 1% 1.91µs ± 4% ~ (p=0.200 n=3+3) > FormatEmoji-8 1.97µs ± 3% 1.99µs ± 1% ~ (p=0.700 n=3+3) > FormatEmoji-12 2.05µs ± 1% 2.03µs ± 1% ~ (p=0.400 n=3+3) > > name old alloc/op new alloc/op delta > FormatEmoji-2 384B ± 0% 384B ± 0% ~ (p=1.000 n=3+3) > FormatEmoji-4 386B ± 0% 385B ± 0% ~ (p=0.700 n=3+3) > FormatEmoji-8 387B ± 0% 386B ± 0% ~ (p=0.400 n=3+3) > FormatEmoji-12 388B ± 0% 388B ± 0% ~ (p=1.000 n=3+3) > > name old allocs/op new allocs/op delta > FormatEmoji-2 7.00 ± 0% 7.00 ± 0% ~ (all equal) > FormatEmoji-4 7.00 ± 0% 7.00 ± 0% ~ (all equal) > FormatEmoji-8 7.00 ± 0% 7.00 ± 0% ~ (all equal) > FormatEmoji-12 7.00 ± 0% 7.00 ± 0% ~ (all equal) > > However, > > Why benchstat thinks there is no differences between the following two > runs? > > $ benchstat /tmp/old.log /tmp/new.log > name old time/op new time/op delta > FormatEmojiString-2 1.79µs ± 0% 6.81µs ± 2% ~ (p=0.100 n=3+3) > FormatEmojiString-4 1.99µs ± 1% 7.22µs ± 1% ~ (p=0.100 n=3+3) > FormatEmojiString-8 1.97µs ± 3% 7.55µs ± 2% ~ (p=0.100 n=3+3) > FormatEmojiString-12 2.05µs ± 1% 7.64µs ± 1% ~ (p=0.100 n=3+3) > > name old alloc/op new alloc/op delta > FormatEmojiString-2 384B ± 0% 1459B ± 0% ~ (p=0.100 n=3+3) > FormatEmojiString-4 386B ± 0% 1463B ± 0% ~ (p=0.100 n=3+3) > FormatEmojiString-8 387B ± 0% 1468B ± 0% ~ (p=0.100 n=3+3) > FormatEmojiString-12 388B ± 0% 1473B ± 0% ~ (p=0.100 n=3+3) > > name old allocs/op new allocs/op delta > FormatEmojiString-2 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3) > FormatEmojiString-4 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3) > FormatEmojiString-8 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3) > FormatEmojiString-12 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3) > > I'd think that the delta will be quite big. > > Now take a look at this. benchstat thinks there is no differences between > the following two runs is beyond imagination: > > $ benchstat /tmp/old.log /tmp/new.log > name old time/op new time/op delta > FormatEmojiBlock-2 1.79µs ± 0% 13543.09µs ± 4% ~ (p=0.100 n=3+3) > FormatEmojiBlock-4 1.99µs ± 1% 14597.09µs ± 1% ~ (p=0.100 n=3+3) > FormatEmojiBlock-8 1.97µs ± 3% 16390.62µs ± 2% ~ (p=0.100 n=3+3) > FormatEmojiBlock-12 2.05µs ± 1% 17409.37µs ± 2% ~ (p=0.100 n=3+3) > > name old alloc/op new alloc/op delta > FormatEmojiBlock-2 384B ± 0% 15270330B ± 0% ~ (p=0.100 n=3+3) > FormatEmojiBlock-4 386B ± 0% 15270671B ± 0% ~ (p=0.100 n=3+3) > FormatEmojiBlock-8 387B ± 0% 15271206B ± 0% ~ (p=0.100 n=3+3) > FormatEmojiBlock-12 388B ± 0% 15271727B ± 0% ~ (p=0.100 n=3+3) > > name old allocs/op new allocs/op delta > FormatEmojiBlock-2 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3) > FormatEmojiBlock-4 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3) > FormatEmojiBlock-8 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3) > FormatEmojiBlock-12 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3) > > What's happening? What I've done wrong? > > Thx > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/a56a17b1-a8c3-4926-a27a-21cd86074f67n%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/a56a17b1-a8c3-4926-a27a-21cd86074f67n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Kurtis Rader Caretaker of the exceptional canines Junior and Hank -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD8R2NBveVArAchBOkrL1qZ6oNRpFqWLqbF7aKG5o3L%2Bxw%40mail.gmail.com.
