Hi Simon, Thank you for the series.
On mer., janv. 15, 2025 at 06:30, Simon Glass <[email protected]> wrote: > The current method of running unit tests relies on subcommands of the > ut command. Only the code in each subcommand knows how to find the tests > related to that subcomand. > > This is not ideal and we now have quite a few subcommands which do > nothing but locate the relevant tests in a linker list, then call a > common function to run them. > > This series adds a list of test suites, so that these subcommands can be > removed. > > An issue with 'ut all' is that it doesn't record how many tests failed > overall, so it is necessary to examine copious amounts of output to look > for failures. This series adds a new 'total' feature allow recording the > total number of failed tests. > > To help with 'ut all' a new pytest is created which runs it (as well as > 'ut info') and makes sure that all is well. Due to the 'ut all' failures > this does not pass, so the test is disabled for now. It is here because > it provides security against misnaming a test suite and causing it not > to run. I wanted to test this, but it seems it does not apply on both master or next: With master: 178f6ecb21fe ("Merge patch series "bootstd: Support recording images"") With next: d6da3dbaef57 ("Merge patch series "cmd: Add support for optee commands."") When running: $ b4 shazam -s -l --check [email protected] I see: """ [...] snip Total patches: 43 --- Applying: test: Drop unused suite prototypes Applying: test: Rename test_get_state() to ut_get_state() Applying: test: Add functions to init and uninit the test state Applying: test: Pass the test-state into ut_run_list() Applying: Improve support for linker lists in data structures Applying: test: Add newlines to hush-test messages Applying: test: Rename test suites to match their linker-list name Applying: test: Update ut info to show suites Applying: test/py: Add a test which runs all unit tests Applying: test: Drop the _test suffix on linker lists Applying: test: Introduce a better array of test suites Patch failed at 0011 test: Introduce a better array of test suites error: sha1 information is lacking or useless (test/cmd_ut.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch hint: When you have resolved this problem, run "git am --continue". hint: If you prefer to skip this patch, run "git am --skip" instead. hint: To restore the original branch and stop patching, run "git am --abort". """ On what base should this be applied if we want to test this? Sorry if this is a sily question, usually I get dependencies from the cover letter but I could not find any here. > > Future work may: > - get 'ut all' passing > - enable test_suite() in CL, to ensure that 'ut all' keeps passing > - record duration of each suite > - allow running the tests in random order to tease out dependencies > - tweak the output to remove common prefixes > - getting rid of bootstd, optee and seame 'ut' subcommands > > > Simon Glass (43): > test: Drop unused suite prototypes > test: Rename test_get_state() to ut_get_state() > test: Add functions to init and uninit the test state > test: Pass the test-state into ut_run_list() > Improve support for linker lists in data structures > test: Add newlines to hush-test messages > test: Rename test suites to match their linker-list name > test: Update ut info to show suites > test/py: Add a test which runs all unit tests > test: Drop the _test suffix on linker lists > test: Introduce a better array of test suites > test: Drop the function for running bdinfo tests > test: Drop the function for running cmd tests > test: Drop the function for running common tests > test: Drop the function for running dm tests > test: Drop the function for running env tests > test: Drop the function for running exit tests > test: Drop the function for running fdt tests > test: Drop the function for running font tests > test: Drop the function for running lib tests > test: Drop the function for running log tests > test: Drop the function for running mbr tests > test: Drop the function for running mem tests > test: Drop the function for running setexpr tests > test: Drop the function for running measurement tests > test: Drop the function for running bloblist tests > test: Drop the function for running bootm tests > test: Drop the function for running addrmap tests > test: Drop the function for running hush tests > test: Drop the function for running loadm tests > test: Drop the function for running pci_mps tests > test: Drop the function for running seama tests > test: Drop the function for running upl tests > test: Drop the info test from the list > test: Drop conditional compilation for suites > test: Pass the test state to cmd_ut_category() > test: Move stats into a struct > test: Keep a track of the numbers of tests run > test: Move stat-printing into its own function > test: Record and show the totals for all test runs > test: Sort the test suites > test: Disable test_suite > test: Move help into the suite declaration > > arch/sandbox/cpu/spl.c | 6 +- > include/dm/test.h | 2 +- > include/linker_lists.h | 51 +++++ > include/test/cmd.h | 2 +- > include/test/common.h | 2 +- > include/test/env.h | 2 +- > include/test/hush.h | 2 +- > include/test/lib.h | 2 +- > include/test/log.h | 4 +- > include/test/optee.h | 2 +- > include/test/overlay.h | 2 +- > include/test/suites.h | 53 ++--- > include/test/test.h | 23 ++- > include/test/ut.h | 45 ++++- > test/boot/bootm.c | 11 +- > test/boot/bootstd_common.c | 9 +- > test/boot/bootstd_common.h | 3 +- > test/boot/measurement.c | 12 +- > test/boot/upl.c | 11 +- > test/cmd/Makefile | 4 +- > test/cmd/addrmap.c | 11 +- > test/cmd/bdinfo.c | 10 +- > test/cmd/cmd_ut_cmd.c | 20 -- > test/cmd/exit.c | 11 +- > test/cmd/fdt.c | 10 +- > test/cmd/font.c | 10 +- > test/cmd/loadm.c | 11 +- > test/cmd/mbr.c | 10 +- > test/cmd/mem.c | 19 -- > test/cmd/mem_copy.c | 2 +- > test/cmd/mem_search.c | 2 +- > test/cmd/pci_mps.c | 12 +- > test/cmd/seama.c | 11 +- > test/cmd/setexpr.c | 11 +- > test/cmd_ut.c | 361 +++++++++++++++++++++------------- > test/common/Makefile | 1 - > test/common/bloblist.c | 12 +- > test/common/cmd_ut_common.c | 21 -- > test/dm/Makefile | 2 - > test/dm/test-dm.c | 16 -- > test/dm/test-driver.c | 10 +- > test/dm/test-uclass.c | 8 +- > test/env/cmd_ut_env.c | 10 - > test/hush/Makefile | 1 - > test/hush/cmd_ut_hush.c | 19 -- > test/hush/dollar.c | 7 +- > test/hush/loop.c | 4 +- > test/lib/Makefile | 1 - > test/lib/cmd_ut_lib.c | 19 -- > test/log/Makefile | 2 - > test/log/log_ut.c | 20 -- > test/optee/cmd_ut_optee.c | 4 +- > test/overlay/cmd_ut_overlay.c | 14 +- > test/py/conftest.py | 7 +- > test/py/tests/test_spl.py | 2 +- > test/py/tests/test_suite.py | 182 +++++++++++++++++ > test/py/tests/test_upl.py | 2 +- > test/py/tests/test_ut.py | 2 +- > test/py/tests/test_vbe.py | 2 +- > test/py/tests/test_vpl.py | 2 +- > test/test-main.c | 104 ++++++---- > test/ut.c | 4 +- > 62 files changed, 657 insertions(+), 580 deletions(-) > delete mode 100644 test/cmd/cmd_ut_cmd.c > delete mode 100644 test/cmd/mem.c > delete mode 100644 test/common/cmd_ut_common.c > delete mode 100644 test/dm/test-dm.c > delete mode 100644 test/hush/cmd_ut_hush.c > delete mode 100644 test/lib/cmd_ut_lib.c > delete mode 100644 test/log/log_ut.c > create mode 100644 test/py/tests/test_suite.py > > -- > 2.34.1
