On 15.01.25 14:30, Simon Glass wrote:
The current method of running unit tests relies on subcommands of the
ut command. Only the code in each subcommand knows how to find the tests
related to that subcomand.
This is not ideal and we now have quite a few subcommands which do
nothing but locate the relevant tests in a linker list, then call a
common function to run them.
This series adds a list of test suites, so that these subcommands can be
removed.
An issue with 'ut all' is that it doesn't record how many tests failed
overall, so it is necessary to examine copious amounts of output to look
for failures. This series adds a new 'total' feature allow recording the
total number of failed tests.
To help with 'ut all' a new pytest is created which runs it (as well as
'ut info') and makes sure that all is well. Due to the 'ut all' failures
this does not pass, so the test is disabled for now. It is here because
it provides security against misnaming a test suite and causing it not
to run.
Future work may:
- get 'ut all' passing
- enable test_suite() in CL, to ensure that 'ut all' keeps passing
- record duration of each suite
- allow running the tests in random order to tease out dependencies
- tweak the output to remove common prefixes
- getting rid of bootstd, optee and seame 'ut' subcommands
Am I missing a patch for doc/ or will the usage of the ut command remain
unchanged?
How would you specify a test suite like 'lib' in future?
How would you specify running all tests in future?
Best regards
Heinrich
Simon Glass (43):
test: Drop unused suite prototypes
test: Rename test_get_state() to ut_get_state()
test: Add functions to init and uninit the test state
test: Pass the test-state into ut_run_list()
Improve support for linker lists in data structures
test: Add newlines to hush-test messages
test: Rename test suites to match their linker-list name
test: Update ut info to show suites
test/py: Add a test which runs all unit tests
test: Drop the _test suffix on linker lists
test: Introduce a better array of test suites
test: Drop the function for running bdinfo tests
test: Drop the function for running cmd tests
test: Drop the function for running common tests
test: Drop the function for running dm tests
test: Drop the function for running env tests
test: Drop the function for running exit tests
test: Drop the function for running fdt tests
test: Drop the function for running font tests
test: Drop the function for running lib tests
test: Drop the function for running log tests
test: Drop the function for running mbr tests
test: Drop the function for running mem tests
test: Drop the function for running setexpr tests
test: Drop the function for running measurement tests
test: Drop the function for running bloblist tests
test: Drop the function for running bootm tests
test: Drop the function for running addrmap tests
test: Drop the function for running hush tests
test: Drop the function for running loadm tests
test: Drop the function for running pci_mps tests
test: Drop the function for running seama tests
test: Drop the function for running upl tests
test: Drop the info test from the list
test: Drop conditional compilation for suites
test: Pass the test state to cmd_ut_category()
test: Move stats into a struct
test: Keep a track of the numbers of tests run
test: Move stat-printing into its own function
test: Record and show the totals for all test runs
test: Sort the test suites
test: Disable test_suite
test: Move help into the suite declaration
arch/sandbox/cpu/spl.c | 6 +-
include/dm/test.h | 2 +-
include/linker_lists.h | 51 +++++
include/test/cmd.h | 2 +-
include/test/common.h | 2 +-
include/test/env.h | 2 +-
include/test/hush.h | 2 +-
include/test/lib.h | 2 +-
include/test/log.h | 4 +-
include/test/optee.h | 2 +-
include/test/overlay.h | 2 +-
include/test/suites.h | 53 ++---
include/test/test.h | 23 ++-
include/test/ut.h | 45 ++++-
test/boot/bootm.c | 11 +-
test/boot/bootstd_common.c | 9 +-
test/boot/bootstd_common.h | 3 +-
test/boot/measurement.c | 12 +-
test/boot/upl.c | 11 +-
test/cmd/Makefile | 4 +-
test/cmd/addrmap.c | 11 +-
test/cmd/bdinfo.c | 10 +-
test/cmd/cmd_ut_cmd.c | 20 --
test/cmd/exit.c | 11 +-
test/cmd/fdt.c | 10 +-
test/cmd/font.c | 10 +-
test/cmd/loadm.c | 11 +-
test/cmd/mbr.c | 10 +-
test/cmd/mem.c | 19 --
test/cmd/mem_copy.c | 2 +-
test/cmd/mem_search.c | 2 +-
test/cmd/pci_mps.c | 12 +-
test/cmd/seama.c | 11 +-
test/cmd/setexpr.c | 11 +-
test/cmd_ut.c | 361 +++++++++++++++++++++-------------
test/common/Makefile | 1 -
test/common/bloblist.c | 12 +-
test/common/cmd_ut_common.c | 21 --
test/dm/Makefile | 2 -
test/dm/test-dm.c | 16 --
test/dm/test-driver.c | 10 +-
test/dm/test-uclass.c | 8 +-
test/env/cmd_ut_env.c | 10 -
test/hush/Makefile | 1 -
test/hush/cmd_ut_hush.c | 19 --
test/hush/dollar.c | 7 +-
test/hush/loop.c | 4 +-
test/lib/Makefile | 1 -
test/lib/cmd_ut_lib.c | 19 --
test/log/Makefile | 2 -
test/log/log_ut.c | 20 --
test/optee/cmd_ut_optee.c | 4 +-
test/overlay/cmd_ut_overlay.c | 14 +-
test/py/conftest.py | 7 +-
test/py/tests/test_spl.py | 2 +-
test/py/tests/test_suite.py | 182 +++++++++++++++++
test/py/tests/test_upl.py | 2 +-
test/py/tests/test_ut.py | 2 +-
test/py/tests/test_vbe.py | 2 +-
test/py/tests/test_vpl.py | 2 +-
test/test-main.c | 104 ++++++----
test/ut.c | 4 +-
62 files changed, 657 insertions(+), 580 deletions(-)
delete mode 100644 test/cmd/cmd_ut_cmd.c
delete mode 100644 test/cmd/mem.c
delete mode 100644 test/common/cmd_ut_common.c
delete mode 100644 test/dm/test-dm.c
delete mode 100644 test/hush/cmd_ut_hush.c
delete mode 100644 test/lib/cmd_ut_lib.c
delete mode 100644 test/log/log_ut.c
create mode 100644 test/py/tests/test_suite.py
