Jeff King <p...@peff.net> writes: > Because perf-lib is built on test-lib, it already reads > GIT-BUILD-OPTIONS. And the Makefile copies several perf-related values > into it, including GIT_PERF_MAKE_OPTS and GIT_PERF_REPEAT_COUNT. So you > can already do: > ... > But right now the perf suite is not useful at > all for comparing two builds of the same tree. > For that, I think it > would be more useful if we could define a tuple of parameters for a run. > One of which could be the tree we're testing. Build opts are another. > Tested repository is another. And then we'd fill in a table of results > and let you slice up the table by any column (e.g., compare times for > runs against a single tree but with differing build options).
Yeah, I think we saw this discussed in not-so-distant past, for which we want a good solution, and it might be the case that such a solution can be made easier to use with a separate configuration file (which this topic may or may not be used as a building block). > So then I think your config file primarily becomes about defining the > properties of each run. I'm not sure if it would look like what you're > starting on here or not. Yeah, I suspect that the final shape that defines the matrix might have to become quite a bit different.
