Andrei Alexandrescu wrote: > On 4/10/12 5:40 AM, Jens Mueller wrote: > >How come that the times based relative report and the percentage based > >relative report are mixed in one result? And how do I choose which one > >I'd like to see in the output. > > It's in the names. If the name of a benchmark starts with > benchmark_relative_, then that benchmark is considered relative to > the last non-relative benchmark. Using a naming convention allows > complete automation in benchmarking a module. > > I figure it's fine that all results appear together because the > absence of data in the relative column clarifies which is which.
I know. That wasn't my question. How do I choose between percentage vs. factors for a relative benchmark, e.g. 200% vs. 2? > >When benchmarking you can measure different things at the same time. In > >this regard the current proposal is limited. It just measures wall clock > >time. I believe extending the StopWatch to measure e.g. user CPU time is > >a useful addition. > > Generally I fear piling too much on StopWatch because every feature > adds its own noise. But there's value in collecting the result of > times(). What would be the Windows equivalent? I'm not a Windows user but GetProcessTimes seems to be the Windows equivalent. http://msdn.microsoft.com/en-us/library/windows/desktop/ms683223%28v=vs.85%29.aspx > >In general, allowing user defined measurements would be great. > >E.g. to measure the time spend in user mode. > >() => { > > tms t; > > times(&t); > > return t.tms_utime; > > } > > > >Note, that this code does not need to be portable. You can also use > >version() else static assert. > > > >Things that come to mind that I'd like to measure. > >Time measurements: > > * User CPU time > > * System CPU time > > * Time spent in memory allocations > >Count measurements: > > * Memory usage > > * L1/L2/L3 cache misses > > * Number of executed instructions > > * Number of memory allocations > > > >Of course wall clock time is the ultimate measure when benchmarking. > >But often you need to investigate further (doing more measurements). > > > >Do you think adding this is worthwhile? > > Absolutely. I just fear about expanding the charter of the framework > too much. Let's see: > > * Memory usage is, I think, difficult in Windows. > * Estimating cache misses and executed instructions is significant research > * Number of memory allocations requires instrumenting druntime I think that std.benchmark shouldn't do this. But I think we should figure out whether adding a user-defined measurement function is possible. Making sure that be we have a design that is flexible enough to capture different measurements. Jens
