> On May 10, 2016, 1:12 a.m., Joe Gross wrote:
> > Hi Andreas, I had a couple questions/clarifications about this patch. The
> > idea is that this script will run the test/test suites in an ordered
> > fashion but doesn't contain any infrastructure to launch jobs in a
> > distributed environment, correct? You mentioned that these work in a CI
> > environment and can be reproduced using only the .pickle files, but it
> > seems that actually job submission is left to users.
> > Also, the reporting is via output files that could be scanned by some other
> > process, but it doesn't actively push results, do I understand that
> > correctly?
> > Lastly, it seems that this is intended to be used once a build or builds
> > have finished, but wouldn't be responsible for building or updating builds,
> > right?
> > It looks like a nice improvement and I just want to be sure I understand
> > the scope and capabilities correctly.
Hi Joe,
Launching test: You're right that the script isn't intended to launch things in
a distributed way. The script supports running multiple tests in series (it's
not very useful in practice, so running multiple tests should probably by
dropped). What I typically do to launch tests is something along these lines
(see RB3462 for the scons side):
```
# Build phase, usually on a separate node
scons build/ARM/gem5.opt
scons build/ARM/tests/opt/all.list
# Test phase. Tests usually run as individual jobs in a cluster.
for F in `cat build/ARM/tests/opt/all.list`; do
mkdir m5tests/$F
submit_cluster_job ./tests/tests.py run --format pickle -o
m5tests/$F/status.pickle build/ARM/gem5.opt $F
done
```
I'm not sure what you mean by pushing results. Is that related to the CI system
integration? I normally do that by merging pickle files into one JUnit file:
```
./tests/tests.py ./tests/tests.py show --format junit -o junit.xml `find
m5tests/ -name status.pickle`
```
The pickle file is then read and formatted by the CI system (which obviously
must be configured to pick up the JUnit output).
In terms of integrating with the build environment, this script doesn't try to
build anything. You point it to a gem5 binary you want to test and tell it
which tests to run. There is a separate scons (RB3462) integration that
supports building as well. In practice, that just implements the old behaviour
using the new infrastructure.
I am considering adding support for updating test references using the test
script. This would be exteremely useful in our CI environment since it stores a
tar ball of the raw test output. This tar ball could be used to update stats if
the nightly runs show that they differ.
I hope that answers all of your questions.
- Andreas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://reviews.gem5.org/r/3461/#review8294
-----------------------------------------------------------
On May 3, 2016, 6:44 p.m., Andreas Sandberg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://reviews.gem5.org/r/3461/
> -----------------------------------------------------------
>
> (Updated May 3, 2016, 6:44 p.m.)
>
>
> Review request for Default.
>
>
> Repository: gem5
>
>
> Description
> -------
>
> Changeset 11472:23a195434229
> ---------------------------
> tests: Add test infrastructure as a Python module
>
> Implement gem5's test infrastructure as a Python module and a run
> script that can be used without scons. The new implementation has
> several features that were lacking from the previous test
> infrastructure such as support for multiple output formats, automatic
> runtime tracking, and better support for being run in a cluster
> environment.
>
> Tests consist of one or more steps (TestUnit). Units are run in two
> stages, the first a run stage and then a verify stage. Units in the
> verify stage are automatically skipped if any unit run stage wasn't
> run. The library currently contains TestUnit implementations that run
> gem5, diff stat files, and diff output files.
>
> Existing tests are implemented by the ClassicTest class and "just
> work". New tests can that don't rely on the old "run gem5 once and
> diff output" strategy can be implemented by subclassing the Test base
> class or ClassicTest.
>
> Test results can be output in multiple formats. The module currently
> supports JUnit, text (short and verbose), and Python's pickle
> format. JUnit output allows CI systems to automatically get more
> information about test failures. The pickled output contains all state
> necessary to reconstruct a tests results object and is mainly intended
> for the build system and CI systems.
>
> Since many JUnit parsers parsers assume that test suite names look
> like Java package names. We currently output path-like names with
> slashes separating components. Test names are translated according to
> these rules:
>
> * '.' -> '-"
> * '/' -> '.'
>
> The test tool, tests.py, supports the following features:
>
> * Test listing. Example: ./tests.py list arm/quick
>
> * Running tests. Example:
> ./tests.py run -o output.pickle --format pickle \
> ../build/ARM/gem5.opt \
> quick/se/00.hello/arm/linux/simple-timing
>
> * Displaying pickled results. Example:
> ./tests.py show --format summary *.pickle
>
> Change-Id: I527164bd791237aacfc65e7d7c0b67b695c5d17c
> Signed-off-by: Andreas Sandberg <[email protected]>
> Reviewed-by: Curtis Dunham <[email protected]>
>
>
> Diffs
> -----
>
> tests/testing/__init__.py PRE-CREATION
> tests/testing/helpers.py PRE-CREATION
> tests/testing/results.py PRE-CREATION
> tests/testing/tests.py PRE-CREATION
> tests/testing/units.py PRE-CREATION
> tests/tests.py PRE-CREATION
>
> Diff: http://reviews.gem5.org/r/3461/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Andreas Sandberg
>
>
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
