> On May 10, 2016, 1:12 a.m., Joe Gross wrote: > > Hi Andreas, I had a couple questions/clarifications about this patch. The > > idea is that this script will run the test/test suites in an ordered > > fashion but doesn't contain any infrastructure to launch jobs in a > > distributed environment, correct? You mentioned that these work in a CI > > environment and can be reproduced using only the .pickle files, but it > > seems that actually job submission is left to users. > > Also, the reporting is via output files that could be scanned by some other > > process, but it doesn't actively push results, do I understand that > > correctly? > > Lastly, it seems that this is intended to be used once a build or builds > > have finished, but wouldn't be responsible for building or updating builds, > > right? > > It looks like a nice improvement and I just want to be sure I understand > > the scope and capabilities correctly. > > Andreas Sandberg wrote: > Hi Joe, > > Launching test: You're right that the script isn't intended to launch > things in a distributed way. The script supports running multiple tests in > series (it's not very useful in practice, so running multiple tests should > probably by dropped). What I typically do to launch tests is something along > these lines (see RB3462 for the scons side): > > ``` > # Build phase, usually on a separate node > scons build/ARM/gem5.opt > scons build/ARM/tests/opt/all.list > > # Test phase. Tests usually run as individual jobs in a cluster. > for F in `cat build/ARM/tests/opt/all.list`; do > mkdir m5tests/$F > submit_cluster_job ./tests/tests.py run --format pickle -o > m5tests/$F/status.pickle build/ARM/gem5.opt $F > done > ``` > > I'm not sure what you mean by pushing results. Is that related to the CI > system integration? I normally do that by merging pickle files into one > JUnit file: > > ``` > ./tests/tests.py ./tests/tests.py show --format junit -o junit.xml `find > m5tests/ -name status.pickle` > ``` > > The pickle file is then read and formatted by the CI system (which > obviously must be configured to pick up the JUnit output). > > In terms of integrating with the build environment, this script doesn't > try to build anything. You point it to a gem5 binary you want to test and > tell it which tests to run. There is a separate scons (RB3462) integration > that supports building as well. In practice, that just implements the old > behaviour using the new infrastructure. > > I am considering adding support for updating test references using the > test script. This would be exteremely useful in our CI environment since it > stores a tar ball of the raw test output. This tar ball could be used to > update stats if the nightly runs show that they differ. > > I hope that answers all of your questions. > > Joe Gross wrote: > Hi Andreas, thanks for your detailed reply. We're currently looking at > leveraging your tester infrastructure for our internal tests which is why > I've had a few questions for you. We'd like to be able to distribute tests to > our job queuing system. Do you think it's easy to optionally run all the > tests in parallel in this way? Also we were looking at how we can test > building various configurations of gem5 and benchmark binaries and running > combinations, do you think this is possible also?
It should be straight forward to do that. That's what we're doing internally. The (all|quick|long).list scons targets are there to make test selection straight forward and ensures you get the exact same tests as in a normal scons run (calling tests.py list ... requires the protocol/gpu parameters to be set manually). The run command blocks until the test completes, so it should be straight forward to execute in your cluster environment. There are no dependencies between jobs and the script ensures that output files end up in different directories, so it should be safe to execue jobs in parallel. Except for the gem5 binary under test, there are no dependencies on build artifacts in the gem5 tree. We typically build and test in different steps in our CI system. In practice, that means that our test (submission) jobs pull in binaries from the build jobs and check out a clean gem5 source tree (same revision as used in the build) that is used for test dependencies (scripts, binaries, etc.). What you need to keep in mind though is that the run and format commands will won't signal their status using exit codes at the moment. This means that the only way to detect failed tests when run like this is by parsing the junit file. I'm planning to fix this though. - Andreas ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://reviews.gem5.org/r/3461/#review8294 ----------------------------------------------------------- On May 3, 2016, 6:44 p.m., Andreas Sandberg wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > http://reviews.gem5.org/r/3461/ > ----------------------------------------------------------- > > (Updated May 3, 2016, 6:44 p.m.) > > > Review request for Default. > > > Repository: gem5 > > > Description > ------- > > Changeset 11472:23a195434229 > --------------------------- > tests: Add test infrastructure as a Python module > > Implement gem5's test infrastructure as a Python module and a run > script that can be used without scons. The new implementation has > several features that were lacking from the previous test > infrastructure such as support for multiple output formats, automatic > runtime tracking, and better support for being run in a cluster > environment. > > Tests consist of one or more steps (TestUnit). Units are run in two > stages, the first a run stage and then a verify stage. Units in the > verify stage are automatically skipped if any unit run stage wasn't > run. The library currently contains TestUnit implementations that run > gem5, diff stat files, and diff output files. > > Existing tests are implemented by the ClassicTest class and "just > work". New tests can that don't rely on the old "run gem5 once and > diff output" strategy can be implemented by subclassing the Test base > class or ClassicTest. > > Test results can be output in multiple formats. The module currently > supports JUnit, text (short and verbose), and Python's pickle > format. JUnit output allows CI systems to automatically get more > information about test failures. The pickled output contains all state > necessary to reconstruct a tests results object and is mainly intended > for the build system and CI systems. > > Since many JUnit parsers parsers assume that test suite names look > like Java package names. We currently output path-like names with > slashes separating components. Test names are translated according to > these rules: > > * '.' -> '-" > * '/' -> '.' > > The test tool, tests.py, supports the following features: > > * Test listing. Example: ./tests.py list arm/quick > > * Running tests. Example: > ./tests.py run -o output.pickle --format pickle \ > ../build/ARM/gem5.opt \ > quick/se/00.hello/arm/linux/simple-timing > > * Displaying pickled results. Example: > ./tests.py show --format summary *.pickle > > Change-Id: I527164bd791237aacfc65e7d7c0b67b695c5d17c > Signed-off-by: Andreas Sandberg <[email protected]> > Reviewed-by: Curtis Dunham <[email protected]> > > > Diffs > ----- > > tests/testing/__init__.py PRE-CREATION > tests/testing/helpers.py PRE-CREATION > tests/testing/results.py PRE-CREATION > tests/testing/tests.py PRE-CREATION > tests/testing/units.py PRE-CREATION > tests/tests.py PRE-CREATION > > Diff: http://reviews.gem5.org/r/3461/diff/ > > > Testing > ------- > > > Thanks, > > Andreas Sandberg > > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
