Hi Andreas, If I haven't said it before, thanks for all the effort you've put in here updating the regressions! I'm glad to see things moving in a positive direction.
For the scons changes, I think I'll just trust you that they work ;). The SConscript files are pretty much inscrutable to me. My only concern is do we really need to keep the scons support? I guess it doesn't hurt anything, but I don't really see how it makes anything easier either. It isn't like using the old regression tester was straightforward.... or even documented in a clear way. Also, could you add a wiki page explaining how to use the new regression tester? Mostly, I would like to see "These are the steps to run the regressions so you have some confidence your new patch doesn't break anything." Granted, that was missing for the old regressions, but I think it would help our users who may be first time patch submitters. Thanks again for all the effort on this! Jason On Thu, May 26, 2016 at 5:37 AM Andreas Sandberg <[email protected]> wrote: > Everyone, > > The test framework has been sitting on review board for a while now. The > framework code has received one ship it from Joel (thanks!), but the Scons > integration has still not been reviewed. I’m planning to push the > framework code today as it doesn’t interfere with anything without the > Scons integration and I plan to push the Scons integration early next week > unless I hear otherwise. > > Cheers, > Andreas > > > On 28/04/2016, 16:55, "gem5-dev on behalf of Andreas Sandberg" > <[email protected] on behalf of [email protected]> wrote: > > >Fellow gem5 Developers, > > > >As a part of a larger gem5 infrastructure refresh within ARM (switch to > >git among other things), we have reworked parts of the test > >infrastructure. One of the main issues we have with the current test > >framework is that it is hard to integrate with our cluster and CI > >environments. > > > >Currently, when we run tests, we have to run them using scons. This is not > >ideal when running tests in a cluster since the resource requirements are > >different for building (one job spanning a large machine) and running > >tests (one job per test). Another problem with the current test > >environment is that the test results are not ideal for a CI system, which > >typically expects JUnit XML (or similar). To address these we rewrote the > >test infrastructure as a Python library: > > > >tests: Add test infrastructure as a Python module [1] > > > >This package supports test discovery, test running, and output formatting. > > Test cases consist of one or more steps (aka test units). Units are run > >in two stages, the first a run stage and then a verify stage. Units in the > >verify stage are automatically skipped if any unit run stage wasn¹t run. > >The library currently contains unit implementations that run > >gem5, diff stat files, and diff output files. Existing tests are > >implemented by the ClassicTest class and ³just work". New tests can that > >don't rely on the old "run gem5 once and diff output" strategy can be > >implemented by subclassing the Test base class or ClassicTest. > > > >Test results can be output in multiple formats. The module currently > >supports JUnit, text (short and verbose), and Python's pickle format. > >JUnit output allows CI systems to automatically get more information about > >test failures. The pickled output contains all state necessary to > >reconstruct a tests results object and is mainly intended for the build > >system and CI systems. > > > >We have integrated the framework with gem5¹s build system: > > > >scons: Use the new test framework from scons [2] > > > > > >The integration maintains the same ³user interface² as the old Scons > >runner. I.e., you can still run ³scons build/ARM/tests/opt/quick². In > >addition to several under-the-hood changes, the build system now supports > >test listing using a special list target. For example: > > > >scons build/ARM/tests/opt/all.list > > > >This makes it possible to run test cases without invoking scons: > > > > > >for T in `cat build/ARM/tests/opt/all.list`; do > > ./tests/tests.py run build/ARM/gem5.opt $T > >Done > > > > > >Since the test script doesn¹t require tests to be run from the root of > >gem5¹s source tree, we want to avoid paths that assume that. The few > >places where that occurs have been fixed by the following patch: > > > >tests: Enable test running outside of gem5's source tree [3] > > > >Looking forward to hear what you think about the new test framework. > > > > > >Thanks, > >Andreas > > > > > >[1] http://reviews.gem5.org/r/3461/ > >[2] http://reviews.gem5.org/r/3462/ > > > >[3] http://reviews.gem5.org/r/3459/ > > > >IMPORTANT NOTICE: The contents of this email and any attachments are > >confidential and may also be privileged. If you are not the intended > >recipient, please notify the sender immediately and do not disclose the > >contents to any other person, use it for any purpose, or store or copy > >the information in any medium. Thank you. > > > >_______________________________________________ > >gem5-dev mailing list > >[email protected] > >http://m5sim.org/mailman/listinfo/gem5-dev > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > -- Jason _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
