Aleksandar Markovic <aleksandar.qemu.de...@gmail.com> writes:
> On Wednesday, September 2, 2020, Alex Bennée <alex.ben...@linaro.org> wrote: > >> >> Ahmed Karaman <ahmedkhaledkara...@gmail.com> writes: >> >> > A nightly performance testing system to monitor any change in QEMU >> > performance across seventeen different targets. >> > >> > The system includes eight different benchmarks to provide a variety >> > of testing workloads. >> > >> > dijkstra_double: >> > Find the shortest path between the source node and all other nodes >> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn >> > distances are double values. The value of n can be specified using >> > the -n flag. The default value is 2000. >> > >> > dijkstra_int32: >> > Find the shortest path between the source node and all other nodes >> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn >> > distances are int32 values. The value of n can be specified using >> > the -n flag. The default value is 2000. >> > >> > matmult_double: >> > Standard matrix multiplication of an n*n matrix of randomly generated >> > double numbers from 0 to 100. The value of n is passed as an argument >> > with the -n flag. The default value is 200. >> > >> > matmult_int32: >> > Standard matrix multiplication of an n*n matrix of randomly generated >> > integer numbers from 0 to 100. The value of n is passed as an >> > argument with the -n flag. The default value is 200. >> > >> > qsort_double: >> > Quick sort of an array of n randomly generated double numbers from 0 >> > to 1000. The value of n is passed as an argument with the -n flag. >> > The default value is 300000. >> > >> > qsort_int32: >> > Quick sort of an array of n randomly generated integer numbers from 0 >> > to 50000000. The value of n is passed as an argument with the -n >> > flag.The default value is 300000. >> > >> > qsort_string: >> > Quick sort of an array of 10000 randomly generated strings of size 8 >> > (including null terminating character). The sort process is repeated >> > n number of times. The value of n is passed as an argument with the >> > -n flag. The default value is 20. >> > >> > search_string: >> > Search for the occurrence of a small string in a much larger random >> > string (“needle in a hay”). The search process is repeated n number >> > of times and each time, a different large random string (“hay”) is >> > generated. The value of n can be specified using the -n flag. The >> > default value is 20. >> > >> > Syntax: >> > nightly_tests_core.py [-h] [-r REF] >> > Optional arguments: >> > -h, --help Show this help message and exit >> > -r REF, --reference REF >> > Reference QEMU version - Default is v5.1.0 >> > Example of usage: >> > nightly_tests_core.py -r v5.1.0 2>log.txt >> > >> > The following report includes detailed setup and execution details >> > of the system: >> > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/ >> QEMU-Nightly-Performance-Tests/ >> > >> > Signed-off-by: Ahmed Karaman <ahmedkhaledkara...@gmail.com> >> > --- >> > tests/performance/nightly-tests/README.md | 243 +++++ >> > .../source/dijkstra_double/dijkstra_double.c | 194 ++++ >> > .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ >> > .../source/matmult_double/matmult_double.c | 123 +++ >> > .../source/matmult_int32/matmult_int32.c | 121 +++ >> > .../source/qsort_double/qsort_double.c | 104 ++ >> > .../source/qsort_int32/qsort_int32.c | 103 ++ >> > .../source/qsort_string/qsort_string.c | 122 +++ >> > .../source/search_string/search_string.c | 110 +++ >> > .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ >> > .../scripts/run_nightly_tests.py | 135 +++ >> > .../nightly-tests/scripts/send_email.py | 56 ++ >> > 12 files changed, 2423 insertions(+) >> > create mode 100644 tests/performance/nightly-tests/README.md >> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ >> dijkstra_double/dijkstra_double.c >> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ >> dijkstra_int32/dijkstra_int32.c >> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ >> matmult_double/matmult_double.c >> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ >> matmult_int32/matmult_int32.c >> > create mode 100644 tests/performance/nightly- >> tests/benchmarks/source/qsort_double/qsort_double.c >> > create mode 100644 tests/performance/nightly- >> tests/benchmarks/source/qsort_int32/qsort_int32.c >> > create mode 100644 tests/performance/nightly- >> tests/benchmarks/source/qsort_string/qsort_string.c >> > create mode 100644 >> > tests/performance/nightly-tests/benchmarks/source/ >> search_string/search_string.c >> >> Perhaps we could compress these paths down to: >> >> tests/tcg/benchmarks/foo.c >> tests/tcg/benchmarks/bar.c >> >> and then we can also ensure they are built using the existing TCG tests >> cross compile framework. >> >> > Hi, > > May I just bring some alternative views on this topic of > benchmarks/multiple cross-compiling/nightly tests? > > Having a known source code of any benchmark (and also license-compatible > with QEMU) is, for sure, a good thing. However, for actual test execution, > source code is not relevant, but the executables built for a diverse set of > targets. Expectation that a test bed should be able to build let's say 17 > version for each benchmark is a tall order - and, in fact, unnecessary > self-imposed limitation. > > My suggestion is that 17x8 executables needed for nightly tests in its > current form should be prebuilt - and kept permanently in a separate > repository (just not to burden QEMU repository). They are anyway built only > once, there is no need to update them at all. > > This will allow nightly test execution on any system that can execute qemu, > without any prerequisite for cross-compiling. > > Their source code could and should be integrated into QEMU tree, at the > place of choice, but the test beds should not be subject to any burden of > cross-compiling, simply because there is no need for that. I have no problem with the test binaries being stored somewhere and accessed when needed - as long as we have a source build-able version in the tree. It would seem silly not to at least integrate the build for the 19 different cross compilers we already support for the check-tcg tests. I'm not proposing we run them every test either, but at least have the option to do: make build-tcg-benchmark make run-tcg-benchmark so it's easy for a developer to get to a state when they can tinker and debug. > > Yours, > Aleksandar > > > > >> > create mode 100755 tests/performance/nightly- >> tests/scripts/nightly_tests_core.py >> > create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_ >> tests.py >> > create mode 100644 tests/performance/nightly- >> tests/scripts/send_email.py >> > >> > diff --git a/tests/performance/nightly-tests/README.md >> b/tests/performance/nightly-tests/README.md >> > new file mode 100644 >> > index 0000000000..6db3b351b3 >> > --- /dev/null >> > +++ b/tests/performance/nightly-tests/README.md >> > @@ -0,0 +1,243 @@ >> > +### QEMU Nightly Tests >> > + >> > +**Required settings:** >> > + >> > +Update the `GMAIL_USER` object in `send_email.py` with your credentials. >> > + >> > +For more details on how the system works, please check the [eighth >> > report](https://ahmedkrmn.github.io/TCG-Continuous- >> Benchmarking/QEMU-Nightly-Performance-Tests/) >> > of the "TCG Continuos Benchmarking" series. >> >> As external URLs are potentially unstable I think we want to distil the >> details into a rst do in docs/devel/ >> >> <snip> >> >> -- >> Alex Bennée >> -- Alex Bennée