OK, that sounds reasonable. We shouldn't have any issues with that.
I'd like someone from us to have a look as well, most likely Giacomo since I won't be able to find the time myself. However, don't let that block anything. Last time I looked at the code, it looked reasonable. I'm sure we can fix any issues once we start integrating it into our test setup. Thanks, Andreas On 03/08/2018 03:26, Sean Wilson wrote: My attempt to address your concerns, Andreas.. Your issue with the baked-in test server was definitely reasonable. I've actually taken that part out of the library since it was so unreliable. A reliable test server would probably be a whole library in itself. My suggestion for running distributed tests would be to do basically the same thing you do now: list all the suites and kick off a distributed task for each suite uid. Each test is individually labeled with a UID. When using the `./main.py run` command you can supply the test suite uid with `--uid <SUITE-UID>`. That test and that test only will run. Your current xml aggregation solution sounds reasonable to me. The only issue you might run into is that a single result.xml is written to, so if all of gem5 is served over NFS you might get conflicts writing to the file. I can add an output path flag if you think this might be an issue for you. Cheers, Sean On Thu, Aug 2, 2018 at 6:21 PM, Jason Lowe-Power <ja...@lowepower.com<mailto:ja...@lowepower.com>> wrote: Hi all, Some more follow up. Sorry that this is going so slowly. Sean just did a pretty major reworking of the main library. See https://gem5-review.googlesource.com/4421. I'll let him address Andreas's questions below. I think we're pretty close to having all three. I would really like to get this pushed in ASAP so someone (me, a student, or others in the community) can start migrating all of the current tests over to the new framework. As a quick preview, this is what the output looks like now.
cd tests ./main.py run
Running the new gem5 testing script. For more information see TESTING.md. To see details as the testing scripts are running, use the option -v, -vv, or -vvv ====================================================================================================================== Loading Tests Discovered 0 tests and 0 suites in /home/jlp/Code/gem5/gem5/tests/tests.py Discovered 30 tests and 15 suites in /home/jlp/Code/gem5/gem5/tests/gem5/hello_se/test_hello_se.py Discovered 6 tests and 3 suites in /home/jlp/Code/gem5/gem5/tests/gem5/m5_util/test_exit.py Discovered 21 tests and 21 suites in /home/jlp/Code/gem5/gem5/tests/gem5/test_build/test_build.py ... <a few errors while we are transitioning> ====================================================================================================================== Running Tests from 9 suites Results will be stored in /home/jlp/Code/gem5/gem5/tests/.testing-results ====================================================================================================================== Building the following targets. This may take a while. /home/jlp/Code/gem5/gem5/build/RISCV/gem5.opt, /home/jlp/Code/gem5/gem5/build/X86/gem5.opt, /home/jlp/Code/gem5/gem5/build/ARM/gem5.opt You may want to run with only a single ISA(--isa=), use --skip-build, or use 'retest'. Test: testhello64-static-X86-opt Passed Test: testhello64-static-X86-opt-MatchStdoutNoPerf Passed Test: testhello64-dynamic-X86-opt Passed Test: testhello64-dynamic-X86-opt-MatchStdoutNoPerf Passed Test: testhello32-static-X86-opt Passed Test: testhello32-static-X86-opt-MatchStdoutNoPerf Passed Test: testhello64-static-ARM-opt Passed Test: testhello64-static-ARM-opt-MatchStdoutNoPerf Passed Test: testhello32-static-ARM-opt Passed Test: testhello32-static-ARM-opt-MatchStdoutNoPerf Passed Test: m5_exit_test-X86-opt Passed Test: m5_exit_test-X86-opt-MatchRegex Passed Test: build-X86-opt Passed Test: build-RISCV-opt Passed Test: build-ARM-opt Passed ====================================== Results: 15 Passed in 2.6e+01 seconds ======================================== Cheers, Jason On Fri, Jul 13, 2018 at 1:37 AM Andreas Sandberg <andreas.sandb...@arm.com<mailto:andreas.sandb...@arm.com>> wrote: Hi Jason, Thanks for doing this. I haven't had a chance to look at the code in details myself, but from what I have seen it looks like a real improvement. Ideally, I'd like someone from our side (probably Giacomo T) to have a look at it. However, Giacomo is currently travelling and I have other pressing matters to tend to. :( I have had a quick look at the code and it all looks reasonable. There are three things we need for CI integration: 1. Ability to separate build and test. 2. Ability to schedule individual (long-running) tests separately in a cluster environment. 3. Ability to get a JUnit XML file with per-test results. With the potential exception of 2, I think the new framework supports all of the above. We could, maybe, use the support for distributed test using the test server in the framework, but I'm a bit reluctant to deploy it for two reasons. 1) It requires credentials to be hard-coded. It would be much more convenient to use an auto-generated authentication file (with suitable permissions) in a shared file location. 2) Having a custom server is potentially very unreliable since the network tends to be noisy (e.g., port scanners, IDS software). The static allocation of a port will also be troublesome (e.g., when two servers run on the same node). In the current framework, we typically list the available tests and then create one cluster job per test. Once the tests complete, the CI framework aggregates all of the JUnit files (it also stores the pickle files for debugging) and generates a report. I'd like to be able to do something with the new framework. Is that possible? Cheers, Andreas On 05/07/2018 21:22, Jason Lowe-Power wrote: Hi all, I'm hoping that we can finalize the new testing framework, and I can push it in within the next week or so. All of the important changes have at least one +2, but I want to give everyone one last chance to object or suggest changes. I've been talking to Sean, and while there's more that could be cleaned up in ext/testlib, we think it's good as it is. At a minimum, this will give us a starting point. I'd like to get this in soon so I can start adding more tests to it in the coming weeks. The patches are linked below: https://gem5-review.googlesource.com/c/public/gem5/+/4880/7 https://gem5-review.googlesource.com/c/public/gem5/+/4881/7 https://gem5-review.googlesource.com/c/public/gem5/+/4421/12 https://gem5-review.googlesource.com/c/public/gem5/+/4882/13 https://gem5-review.googlesource.com/c/public/gem5/+/4883/13 https://gem5-review.googlesource.com/c/public/gem5/+/4422/13 https://gem5-review.googlesource.com/c/public/gem5/+/4423/13 https://gem5-review.googlesource.com/c/public/gem5/+/10121/4 Thanks, Jason _______________________________________________ gem5-dev mailing list gem5-dev@gem5.org<mailto:gem5-dev@gem5.org> http://m5sim.org/mailman/listinfo/gem5-dev IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
