Hi all, Some more follow up. Sorry that this is going so slowly.
Sean just did a pretty major reworking of the main library. See https://gem5-review.googlesource.com/4421. I'll let him address Andreas's questions below. I think we're pretty close to having all three. I would really like to get this pushed in ASAP so someone (me, a student, or others in the community) can start migrating all of the current tests over to the new framework. As a quick preview, this is what the output looks like now. > cd tests > ./main.py run Running the new gem5 testing script. For more information see TESTING.md. To see details as the testing scripts are running, use the option -v, -vv, or -vvv ====================================================================================================================== Loading Tests Discovered 0 tests and 0 suites in /home/jlp/Code/gem5/gem5/tests/tests.py Discovered 30 tests and 15 suites in /home/jlp/Code/gem5/gem5/tests/gem5/hello_se/test_hello_se.py Discovered 6 tests and 3 suites in /home/jlp/Code/gem5/gem5/tests/gem5/m5_util/test_exit.py Discovered 21 tests and 21 suites in /home/jlp/Code/gem5/gem5/tests/gem5/test_build/test_build.py ... <a few errors while we are transitioning> ====================================================================================================================== Running Tests from 9 suites Results will be stored in /home/jlp/Code/gem5/gem5/tests/.testing-results ====================================================================================================================== Building the following targets. This may take a while. /home/jlp/Code/gem5/gem5/build/RISCV/gem5.opt, /home/jlp/Code/gem5/gem5/build/X86/gem5.opt, /home/jlp/Code/gem5/gem5/build/ARM/gem5.opt You may want to run with only a single ISA(--isa=), use --skip-build, or use 'retest'. Test: testhello64-static-X86-opt Passed Test: testhello64-static-X86-opt-MatchStdoutNoPerf Passed Test: testhello64-dynamic-X86-opt Passed Test: testhello64-dynamic-X86-opt-MatchStdoutNoPerf Passed Test: testhello32-static-X86-opt Passed Test: testhello32-static-X86-opt-MatchStdoutNoPerf Passed Test: testhello64-static-ARM-opt Passed Test: testhello64-static-ARM-opt-MatchStdoutNoPerf Passed Test: testhello32-static-ARM-opt Passed Test: testhello32-static-ARM-opt-MatchStdoutNoPerf Passed Test: m5_exit_test-X86-opt Passed Test: m5_exit_test-X86-opt-MatchRegex Passed Test: build-X86-opt Passed Test: build-RISCV-opt Passed Test: build-ARM-opt Passed ====================================== Results: 15 Passed in 2.6e+01 seconds ======================================== Cheers, Jason On Fri, Jul 13, 2018 at 1:37 AM Andreas Sandberg <andreas.sandb...@arm.com> wrote: > Hi Jason, > > Thanks for doing this. I haven't had a chance to look at the code in > details myself, but from what I have seen it looks like a real improvement. > Ideally, I'd like someone from our side (probably Giacomo T) to have a look > at it. However, Giacomo is currently travelling and I have other pressing > matters to tend to. :( > > I have had a quick look at the code and it all looks reasonable. There are > three things we need for CI integration: > > 1. Ability to separate build and test. > 2. Ability to schedule individual (long-running) tests separately in a > cluster environment. > 3. Ability to get a JUnit XML file with per-test results. > > > With the potential exception of 2, I think the new framework supports all > of the above. We could, maybe, use the support for distributed test using > the test server in the framework, but I'm a bit reluctant to deploy it for > two reasons. 1) It requires credentials to be hard-coded. It would be much > more convenient to use an auto-generated authentication file (with suitable > permissions) in a shared file location. 2) Having a custom server is > potentially very unreliable since the network tends to be noisy (e.g., port > scanners, IDS software). The static allocation of a port will also be > troublesome (e.g., when two servers run on the same node). > > In the current framework, we typically list the available tests and then > create one cluster job per test. Once the tests complete, the CI framework > aggregates all of the JUnit files (it also stores the pickle files for > debugging) and generates a report. I'd like to be able to do something with > the new framework. Is that possible? > > Cheers, > Andreas > > On 05/07/2018 21:22, Jason Lowe-Power wrote: > > Hi all, > > I'm hoping that we can finalize the new testing framework, and I can push > it in within the next week or so. All of the important changes have at > least one +2, but I want to give everyone one last chance to object or > suggest changes. I've been talking to Sean, and while there's more that > could be cleaned up in ext/testlib, we think it's good as it is. At a > minimum, this will give us a starting point. I'd like to get this in soon > so I can start adding more tests to it in the coming weeks. > > The patches are linked below: > https://gem5-review.googlesource.com/c/public/gem5/+/4880/7https://gem5-review.googlesource.com/c/public/gem5/+/4881/7https://gem5-review.googlesource.com/c/public/gem5/+/4421/12https://gem5-review.googlesource.com/c/public/gem5/+/4882/13https://gem5-review.googlesource.com/c/public/gem5/+/4883/13https://gem5-review.googlesource.com/c/public/gem5/+/4422/13https://gem5-review.googlesource.com/c/public/gem5/+/4423/13https://gem5-review.googlesource.com/c/public/gem5/+/10121/4 > > Thanks, > Jason > _______________________________________________ > gem5-dev mailing > listgem5-dev@gem5.orghttp://m5sim.org/mailman/listinfo/gem5-dev > > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
