Contains the ReST file for documentation for rtems-tester. Also contains the new format for the ini configuration files. --- user/tools/rtems-tester.rst | 639 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 639 insertions(+) create mode 100644 user/tools/rtems-tester.rst
diff --git a/user/tools/rtems-tester.rst b/user/tools/rtems-tester.rst new file mode 100644 index 0000000..24c62dd --- /dev/null +++ b/user/tools/rtems-tester.rst @@ -0,0 +1,639 @@ +RTEMS Tester +************ + +.. image:: ../../images/user/rtemswhitebg.jpg + :width: 400 + :align: center + +Chris Johns <chr...@rtems.org> +1.0, June 2014 + +RTEMS Tester +============ + +The RTEMS Tester is a test framework. It includes a command line interface to +run tests on supported targets. The framework provides back-end support for +common simulators and debuggers. The board support package (BSP) configurations +for RTEMS are provided and can be used to run all the tests provided with +RTEMS. The framework is not specific to RTEMS and can be configured to run any +suitable application. + +RTEMS is an embedded operating system and is cross-compiled on a range of host +machines. The executables run on the target hardware and this can vary widely +from open source simulators, commercial simulators, debuggers with simulators, +to debuggers with hardware specific pods and devices. Testing RTEMS requires +the cross-compiled test executable is transferred to the target hardware, +executed and the output returned to the host where it is analyzed to determine +the test result. The RTEMS Tester provides a framework to do this. + +Running all the RTEMS tests on your target is very important. It provides you +with a traceable record showing that your RTEMS version and its tools are +working at the level the RTEMS development team expect when releasing +RTEMS. Being able to easily run the tests and verify the results is critical in +maintaining a high standard. + +The RTEMS Tester contains: + +* Command line tool (``rtems-test``) +* BSP Configuration scripts +* Back-end Configuration scripts +* Back-end Python classes +* Python based framework + +This document is for users of the RTEMS Tester as well as those wanting to +develop, extend and investigate the RTEMS Tester. + +License +======= + +The RTEMS Tester is part of the RTEMS Tools Project. The code is released under +the OSI approved "The BSD 2-Clause License". It is free to use and we encourage +this, including on operating systems other than RTEMS. + +The code and command line tools must retain the same names and always reference +the RTEMS Tools Project. + +Quick Start +=========== + +The quick start will show you how to run the test suite for a BSP. It will +explain how to get the RTEMS Tester, set it up and run the tests for the erc32 +BSP. It assumes you have a valid SPARC tool chain and have built the erc32 BSP +version of RTEMS 4.12. + +Setup +===== + +Set up a development work space: + +.. code-block:: shell + + $ cd + $ mkdir -p development/rtems/test + $ cd development/rtems/test + +First fetch the RTEMS tester from the RTEMS Tools repository: + +.. code-block:: shell + + $ git clone git://git.rtems.org/rtems-tools.git rtems-tools.git + $ cd rtems-tools.git/tester + +Available BSP testers +===================== + +You can list the available BSP testers with: + +.. code-block:: shell + + $ ./rtems-test --list-bsps + arm920 + beagleboardxm + jmr3904-run + jmr3904 + mcf5235 + psim-run + psim + realview_pbx_a9_qemu + sis-run + sis + xilinx_zynq_a9_qemu + xilinx_zynq_a9_qemu_smp + xilinx_zynq_zc706 + xilinx_zynq_zc706_qemu + +.. note:: The list is growing all the time and if your BSP is not supported we + encourage you to add it and submit the configuration back to the project. + +Some of the BSPs may appear more than once in the list. These are aliased BSP +configurations that may use a different back-end. An example is the SPARC +Instruction Simulator (SIS) BSP. There is the 'sis' tester which uses the GDB +back-end and the 'sis-run' tester which uses the command line version of the SIS +simulator. We will show how to use ``rtems-test`` command with the SIS BSP +because it is easy to build an use. + +Adding support for a BSP to the tester +====================================== + +To add support for a BSP, one should add a configuration file in +``rtems-tools.git/tester/rtems/testing/bsps``. The configuration files +should be specified in an INI file. The configuration files provide a +means to the backend of `rtems-tester` to automate the running of tests. +There is the ``<bsp>`` tester which uses the GDB back-end and the ``<bsp>-run`` +tester which uses the command line version of the respective simulator. +The general format for specifying an INI configuration file is as follows: + +<bsp>.ini: + +.. code-block:: shell + + [<bsp_name>] + bsp = '<bsp_name>' + gdb = '%{_rtscripts}/gdb.cfg' + arch = '<arch_name>' + + [gdb_script] + gdb_script ='<bsp>_gdb_script' + <bsp>_gdb_script = '<Instructions for gdb + ... + ...>' + +<bsp>-run.ini: + +.. code-block:: shell + + [<bsp_name>] + bsp = '<bsp_name>' + + [run] + run = '%{_rtscripts}/run.cfg' + arch = '<arch_name>' + bsp_run_cmd = '%{rtems_tools}/%{bsp_arch}-rtems%{rtems_version}-run' + bsp_run_opts = '<options>' + +Here is an example: + +arm7tdmi.ini: + +.. code-block:: shell + + [arm7tdmi] + bsp = 'arm7tdmi' + gdb = '%{_rtscripts}/gdb.cfg' + arch = 'arm' + + [gdb_script] + gdb_script ='arm7tdmi_gdb_script' + arm7tdmi_gdb_script = 'target sim + load + run' + +arm7tdmi-run.ini: + +.. code-block:: shell + + [arm7tdmi] + bsp = 'arm7tdmi' + + [run] + run = '%{_rtscripts}/run.cfg' + arch = 'arm' + bsp_run_cmd = '%{rtems_tools}/%{bsp_arch}-rtems%{rtems_version}-run' + bsp_run_opts = '-a -nouartrx' + +If the tester should run the tests on qemu for a bsp, the given pattern can +be followed: + +.. code-block:: shell + + [<bsp_name>] + bsp = '<bsp_name>' + + [qemu-script] + run = '%{_rtscripts}/qemu.cfg' + arch = '<arch_name>' + opts = '%{qemu_opts_base} %{qemu_opts_no_net} -m 32M' + +For instance, + +generic_or1k.ini: + +.. code-block:: shell + + [generic_or1k] + bsp = 'generic_or1k' + + [qemu-script] + run = '%{_rtscripts}/qemu.cfg' + arch = 'or32' + opts = '%{qemu_opts_base} %{qemu_opts_no_net} -m 32M' + +User can also provide a "settings.ini" file in case a configuration file needs +a path to, for instance, a first stage bootloader (fsbl) that is placed +somewhere in the host machine. The settings.ini file can be passed with +``rtems-test --with-settings = /path/to/settings/file`` option. The +<bsp_name>.ini can contain defaults and the settings file can contain the +user-specified configuration. + + +xilinx_zynq_zc706.ini: + +.. code-block:: shell + + [<xilinx_zynq_zc706>] + bsp = 'xilinx_zynq_zc706' + jobs = '1' + + [gdb-script] + gdb = '%{_rtscripts}/gdb.cfg' + arch = 'arm' + bsp_tty_dev = ${settings:bsp_tty_dev} + gdb_script = 'xilinx_zynq_zc706_gdb_script' + xilinx_zynq_zc706_gdb_script = 'target remote kaka:3333 + mon load_image ${settings:path_to_fsbl} 0 elf + mon resume 0 + mon sleep 4000 + mon halt + load + b bsp_reset + continue' + +settings.ini: + +.. code-block:: shell + + [settings] + bsp_tty_dev = '/dev/cu.SLAB_USBtoUART' + path_to_fsbl = '/home/chris/development/si/work/hydra/boot/xilinx-zynq-fsbl/build/arm-rtems4.11-xilinx_zynq_zc706/hydra-fsbl.elf' + +Building RTEMS Tests +==================== + +Build RTEMS with a configuration command line something similar to: + +.. note:: The following assumes a Unix-type host and that the tools have been + built with a prefix of ``$HOME/development/rtems/4.12``. + +.. code-block:: shell + + $ cd + $ export PATH=$HOME/development/rtems/4.12/bin:$PATH + $ mkdir -p development/rtems/kernel + $ cd development/rtems/kernel + $ git clone git://git.rtems.org/rtems.git rtems.git + $ cd rtems.git + $ ./bootstrap -c && ./bootstrap -p && ./bootstrap + $ cd .. + $ mkdir -p builds/erc32 + $ cd builds/erc32 + $ ../../rtems.git/configure --target=sparc-rtems4.12 \ + --enable-tests --enable-rtemsbsp=erc32 + $ make + +Add the -j option to the make command with the number of cores to run the +build parallel where it can. + +Building all the tests takes time and it uses more disk so be patient. When +finished all the tests will be built and ready to run. Before running all the +tests it is a good idea to run the ``hello`` test. The ``hello`` test is an +RTEMS version of the classic "Hello World" example and running it shows you +have a working tool chain and build of RTEMS ready to run the tests. Using the +run command: + +.. code-block:: shell + + $ sparc-rtems4.12-run sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe + + *** BEGIN OF TEST HELLO WORLD *** + Hello World + *** END OF TEST HELLO WORLD *** + +The run command is the GDB simulator without the GDB part. + +Running the example using GDB: + +.. code-block:: shell + + $ sparc-rtems4.12-gdb sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe + GNU gdb (GDB) 7.12 + Copyright (C) 2016 Free Software Foundation, Inc. + License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> + This is free software: you are free to change and redistribute it. + There is NO WARRANTY, to the extent permitted by law. Type "show copying" + and "show warranty" for details. + This GDB was configured as "--host=x86_64-linux-gnu --target=sparc-rtems4.12". + Type "show configuration" for configuration details. + For bug reporting instructions, please see: + <http://www.gnu.org/software/gdb/bugs/>. + Find the GDB manual and other documentation resources online at: + <http://www.gnu.org/software/gdb/documentation/>. + For help, type "help". + Type "apropos word" to search for commands related to "word"... + Reading symbols from + sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe...done. + (gdb) target sim + Connected to the simulator. + (gdb) load + (gdb) r + Starting program: sparc-rtems4.12/c/erc32/testsuites/samples/hello/hello.exe + + + *** BEGIN OF TEST HELLO WORLD *** + Hello World + *** END OF TEST HELLO WORLD *** + [Inferior 1 (process 42000) exited normally] + (gdb) q + +The command r is used to debug set break points before issuing the GDB run +command. + +There are currently close to 500 separate tests and you can run them all with a +single RTEMS Tester command. + +.. note:: Use a recent RSB version to build a SPARC tool chain. A recent patch + fixes issues with the GDB simulator and it improves the test results. + It is also recommended you use a recent RTEMS 4.11 version as the + tests have been cleaned up to improve the results. + +Running the Tests +================= + +The ``rtems-test`` command line accepts a range of options. These are discussed +later in the manual. Any command line argument without a `--` prefix is a test +executable. You can pass more than one executable on the command line. If the +executable is a path to a directory the directories under that path are +searched for any file with a ``.exe`` extension. This is the default extension +for RTEMS executables built within RTEMS. + +To run the erc32 tests enter the following command from the top of the erc32 BSP +build tree: + +.. code-block:: shell + + $ ~/development/rtems/test/rtems-tools.git/tester/rtems-test \ + --log=log_erc32_run \ + --rtems-bsp=erc32-run \ + --rtems-tools=$HOME/development/rtems/4.12 \ + sparc-rtems4.12/c/erc32/testsuites/samples + RTEMS Testing - Tester, 4.12.not_released + [ 1/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: base_sp.exe + [ 2/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: capture.exe + [ 3/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: cdtest.exe + [ 4/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: fileio.exe + [ 5/13] p:2 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: hello.exe + [ 6/13] p:2 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: cxx_iostream.exe + [ 8/13] p:2 f:0 u:0 e:0 I:0 B:0 t:2 i:0 | sparc/erc32: minimum.exe + [ 7/13] p:2 f:0 u:0 e:0 I:0 B:0 t:2 i:0 | sparc/erc32: loopback.exe + [ 9/13] p:3 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: nsecs.exe + [10/13] p:3 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: paranoia.exe + [11/13] p:4 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: pppd.exe + [12/13] p:6 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: ticker.exe + [13/13] p:6 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: unlimited.exe + Passed: 7 + Failed: 0 + User Input: 0 + Expected Fail: 0 + Indeterminate: 0 + Benchmark: 0 + Timeout: 5 + Invalid: 1 + Total: 13 + Average test time: 0:00:27.963000 + Testing time : 0:06:03.519012 + + + +* The RTEMS Tester's test command. In this example we are using an absolute + path. +* The ``--log`` option sends the output to a log file. By default only failed + tests log the complete output. +* Select the erc32 BSP and use GDB. +* Path to the RTEMS tools so GDB can be found. +* Path to the erc32 BSP built with all tests to run. If you add subdirectories + to the path specific tests can be run. +* The output has been shortened so it fits nicely here. +* The test results shows passes, fails, timeouts, and invalid results. In + this run 13 tests passed and 5 tests timed out and 1 is invalid. The + timeouts are probably due to the tests not having enough execute time to + complete. The default timeout is 180 seconds and some of the interrupt tests + need longer. The amount of time depends on the performance of your host CPU + running the simulations. +* The output shows the average time per test and the total time taken to run + all the tests. +* If the path to the testsuites was put to + ``sparc-rtems4.12/c/erc32/testsuites`` instead of + ``sparc-rtems4.12/c/erc32/testsuites/samples`` then all the executables + would have been tested and not just those in samples. + +This BSP requires the ``--rtems-tools`` option because the SPARC GDB is the +``sparc-rtems4.11-gdb`` command that is part of the RTEMS tools. Not every BSP +will require this option so you will need to check the specifics of the BSP +configuration to determine if it is needed. + +The output you see is each test starting to run. The ``rtems-test`` command by +default runs multiple tests in parallel so you will see a number start quickly +and then new tests start as others finish. The output shown here is from an +8 core processor so the first 8 are started in parallel and the status shows +the order in which they actually started, which is not 1 to 8. + +The test start line shows the current status of the tests. The status reported +is when the test starts and not the result of that test. A fail, timeout or +invalid count changing means a test running before this test started failed, +not the starting test. The status here has 7 tests passed, no failures, 5 +timeouts and 1 invalid test. + +.. code-block:: shell + + [ 5/13] p:2 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: hello.exe + +* [ 5/13] indicates the test number, in this case test 5 of 13 tests. +* p is the passed test count (2 in this case) +* f is the failed test count (0 in this case) +* u is the count for test marked as "user-input" as they expect input from user +* e is the expected-fail count (tests that are expected to fail) +* I is the count for tests the results of which are indeterminate +* B is the count for benchmarked tests +* t is the timeout test count +* i is the invalid test count. +* sparc/erc32 is the architecture and BSP names. +* hello.exe is the executable name. + +The test log records all the tests and results. The reporting mode by default +only provides the output history if a test fails, times out, or is invalid. The +time taken by each test is also recorded. + +The tests must complete in a specified time or the test is marked as timed +out. The default timeout is 3 minutes and can be globally changed using the +``--timeout`` command line option. The time required to complete a test can +vary. When simulators are run in parallel the time taken depends on the +specifics of the host machine being used. A test per core is the most stable +method even though more tests can be run than available cores. If your machine +needs longer or you are using a VM you may need to lengthen the timeout. + +Test Status +=========== + +Tests can be marked with one of the following: + +* Pass +* Fail +* Timeout +* Invalid + +The RTEMS console or stdout output from the test is needed to determine the +result of the test. + +Pass +---- +A test passes if the start and end markers are seen in the test output. The +start marker is ``***`` and the end mark is ``*** END OF TEST``. All tests in +the RTEMS test suite have these markers. + +Fail +---- +A test fails if the start marker is seen and there is no end marker. + +Timeout +------- +If the test does not complete within the timeout setting the test is marked as +having timed out. + +Invalid +------- +If no start marker is seen the test is marked as invalid. If you are testing on +real target hardware things can sometimes go wrong and the target may not +initialize or respond to the debugger in an expected way. + +Reporting +========= + +The report written to the log has the following modes: + +* All (``all``) +* Failures (``failures``) +* None (``none``) + +The mode is controlled using the command line option ``--report-mode`` using +the values listed above. + +All +--- +The output of all tests is written to the log. + +Failures +-------- +The output of the all tests that do not pass is written to the log. + +None +---- +No output is written to the log. + +The output is tagged so you can determine where it comes from. The following is +the complete output for the In Memory File System test ``imfs_fslink.exe`` +running on a Coldfire MCF5235 using GDB and a BDM pod: + +.. code-block:: shell + + [ 11/472] p:9 f:0 t:0 i:1 | m68k/mcf5235: imfs_fslink.exe + > gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe + > Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe... + > done. + > target remote | m68k-bdm-gdbserver pipe 003-005 + > Remote debugging using | m68k-bdm-gdbserver pipe 003-005 + > m68k-bdm: debug module version 0 + > m68k-bdm: detected MCF5235 + > m68k-bdm: architecture CF5235 connected to 003-005 + > m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282) + > Process 003-005 created; pid = 0 + > 0x00006200 in ?? () + > thb *0xffe254c0 + > Hardware assisted breakpoint 1 at 0xffe254c0 + > continue + > Continuing. + ] + ] + ] External Reset + ] + ] ColdFire MCF5235 on the BCC + ] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28) + ] Copyright 1995-2004 Freescale Semiconductor, Inc. All Rights Reserved. + ] + ] Enter 'help' for help. + ] + > Temporary breakpoint + > 1, 0xffe254c0 in ?? () + > load + > Loading section .text, size 0x147e0 lma 0x40000 + > Loading section .data, size 0x5d0 lma 0x547e0 + > Start address 0x40414, load size 85424 + > Transfer rate: 10 KB/sec, 1898 bytes/write. + > b bsp_reset + > Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14. + > continue + > Continuing. + ] dBUG> + ] + ] *** FILE SYSTEM TEST ( IMFS ) *** + ] Initializing filesystem IMFS + ] + ] + ] *** LINK TEST *** + ] link creates hardlinks + ] test if the stat is the same + ] chmod and chown + ] unlink then stat the file + ] *** END OF LINK TEST *** + ] + ] + ] Shutting down filesystem IMFS + ] *** END OF FILE SYSTEM TEST ( IMFS ) *** + > Breakpoint + > 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14 + > 14 { + Result: passed Time: 0:00:10.045447 + +* GDB command line (Note: paths with \'....' have been shortened) +* Lines starting with ``>`` are from GDB's console. +* Line starting with ``]`` are from the target's console. +* The result with the test time. + +Running Tests in Parallel +========================= + +The RTEMS Tester supports parallel execution of tests by default. This only +makes sense if the test back-end can run in parallel without resulting in +resource contention. Simulators are an example of back-ends that can run in +parallel. A hardware debug tool like a BDM or JTAG pod can manage only a +single test at once so the tests need to be run one at a time. + +The test framework manages the test jobs and orders the output in the report +log in test order. Output is held for completed tests until the next test to be +reported has finished. + +Command Line Help +================= + +The ``rtems-test`` command line accepts a range of options. You can review the +available option by the ``--help`` option: + +.. code-block:: shell + + RTEMS Tools Project (c) 2012-2014 Chris Johns + Options and arguments: + --always-clean : Always clean the build tree, even with an error + --debug-trace : Debug trace based on specific flags + --dry-run : Do everything but actually run the build + --force : Force the build to proceed + --jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs. + --keep-going : Do not stop on an error. + --list-bsps : List the supported BSPs + --log file : Log file where all build output is written to + --macros file[,file] : Macro format files to load after the defaults + --no-clean : Do not clean up the build tree + --quiet : Quiet output (not used) + --report-mode : Reporting modes, failures (default),all,none + --rtems-bsp : The RTEMS BSP to run the test on + --rtems-tools : The path to the RTEMS tools + --target : Set the target triplet + --timeout : Set the test timeout in seconds (default 180 seconds) + --trace : Trace the execution + --warn-all : Generate warnings + +Development +=========== + +The RTEMS Tester framework and command line tool is under active +development. These are changing, being fixed, broken and generally improved. If +you want to help please see the Wiki page for open items. + +History +======= + +The RTEMS Tester is based on a refactored base of Python code used in the RTEMS +Source Builder. This code provided a working tested base that has been extended +and expanded to meet the requirements for the RTEMS Tester. The tester uses the +specifics found in the various scripts and configurations in the +rtems-testing.git repo that has been accumulated over many years. The shell +script implementation is restricted in what it can do and, per BSP script, is a +maintenance burden. For example the command lines and options vary between each +script. -- 2.1.4 _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel