Fellow gem5 Developers,
As a part of a larger gem5 infrastructure refresh within ARM (switch to
git among other things), we have reworked parts of the test
infrastructure. One of the main issues we have with the current test
framework is that it is hard to integrate with our cluster and CI
environments.
Currently, when we run tests, we have to run them using scons. This is not
ideal when running tests in a cluster since the resource requirements are
different for building (one job spanning a large machine) and running
tests (one job per test). Another problem with the current test
environment is that the test results are not ideal for a CI system, which
typically expects JUnit XML (or similar). To address these we rewrote the
test infrastructure as a Python library:
tests: Add test infrastructure as a Python module [1]
This package supports test discovery, test running, and output formatting.
Test cases consist of one or more steps (aka test units). Units are run
in two stages, the first a run stage and then a verify stage. Units in the
verify stage are automatically skipped if any unit run stage wasn¹t run.
The library currently contains unit implementations that run
gem5, diff stat files, and diff output files. Existing tests are
implemented by the ClassicTest class and ³just work". New tests can that
don't rely on the old "run gem5 once and diff output" strategy can be
implemented by subclassing the Test base class or ClassicTest.
Test results can be output in multiple formats. The module currently
supports JUnit, text (short and verbose), and Python's pickle format.
JUnit output allows CI systems to automatically get more information about
test failures. The pickled output contains all state necessary to
reconstruct a tests results object and is mainly intended for the build
system and CI systems.
We have integrated the framework with gem5¹s build system:
scons: Use the new test framework from scons [2]
The integration maintains the same ³user interface² as the old Scons
runner. I.e., you can still run ³scons build/ARM/tests/opt/quick². In
addition to several under-the-hood changes, the build system now supports
test listing using a special list target. For example:
scons build/ARM/tests/opt/all.list
This makes it possible to run test cases without invoking scons:
for T in `cat build/ARM/tests/opt/all.list`; do
./tests/tests.py run build/ARM/gem5.opt $T
Done
Since the test script doesn¹t require tests to be run from the root of
gem5¹s source tree, we want to avoid paths that assume that. The few
places where that occurs have been fixed by the following patch:
tests: Enable test running outside of gem5's source tree [3]
Looking forward to hear what you think about the new test framework.
Thanks,
Andreas
[1] http://reviews.gem5.org/r/3461/
[2] http://reviews.gem5.org/r/3462/
[3] http://reviews.gem5.org/r/3459/
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the contents to any
other person, use it for any purpose, or store or copy the information in any
medium. Thank you.
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
