Hi Heinrich, On Thu, 4 Mar 2021 at 11:13, Heinrich Schuchardt <xypron.g...@gmx.de> wrote: > > On 3/4/21 2:50 PM, Simon Glass wrote: > > Add details about how to run a sandbox test directly, without using > > pytest. This is more convenient for rapid development, since it is faster > > and allows easier use of a debugger. Also mention sandbox_flattree as an > > example of the different sandbox builds available. > > > > Signed-off-by: Simon Glass <s...@chromium.org> > > --- > > > > (no changes since v2) > > > > Changes in v2: > > - Put the docs in tests_sandbox since it is more related to sandbox > > - Put in a mention of tests_sandbox in the main testing docs > > > > doc/develop/index.rst | 1 + > > doc/develop/testing.rst | 9 ++++ > > doc/develop/tests_sandbox.rst | 79 +++++++++++++++++++++++++++++++++++ > > Thank you for adding this documentation. > > tests_sandbox.rst seems to be the wrong file name:
But if you look at it, it is all about running tests using sandbox. > > We have three types of tests: > > * Python tests > * library tests started using the 'ut' shell commad > * UEFI related tests started using the 'bootefi selftest' command > > All three tpyes of tests are run both on real hardware and on the sandbox. > > Unfortunately a few Python tests require the sandbox. > > Some tests are not run on the sandbox by default though they could, e.g. > the UEFI tests run only on QEMU because you decided not to activate > CONFIG_EFI_SELFTEST on sandbox_defconfig. It was a long time ago, but I recall spending a year trying to upstream a bit of refactoring to allow it and then giving up. If you have a way to do it, please go ahead. > > Some tests can only be used interactively, e.g. > > setenv efi_selftest block image transfer > bootefi selftest > > Some tests cannot be run on the sandbox at all, e.g. > test/py/tests/test_fpga.py That is a bug :-) > > > 3 files changed, 89 insertions(+) > > create mode 100644 doc/develop/tests_sandbox.rst > > > > diff --git a/doc/develop/index.rst b/doc/develop/index.rst > > index ac57fdb8f30..50b1de3bdff 100644 > > --- a/doc/develop/index.rst > > +++ b/doc/develop/index.rst > > @@ -33,3 +33,4 @@ Testing > > coccinelle > > testing > > py_testing > > + tests_sandbox > > diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst > > index f01ca4dc408..87c90eee271 100644 > > --- a/doc/develop/testing.rst > > +++ b/doc/develop/testing.rst > > @@ -36,6 +36,7 @@ U-Boot can be built as a user-space application (e.g. for > > Linux). This > > allows test to be executed without needing target hardware. The 'sandbox' > > target provides this feature and it is widely used in tests. > > > > +See :doc:`tests_sandbox` for more information. > > > > Pytest Suite > > ------------ > > @@ -51,8 +52,16 @@ You can run the tests on sandbox with:: > > > > This will produce HTML output in build-sandbox/test-log.html > > > > +Some tests run with other versions of sandbox. For example sandbox_flattree > > +runs the tests with livetree (the hierachical devicetree) disabled. You can > > +also select particular tests with -k:: > > + > > + ./test/py/test.py --bd sandbox_flattree --build -k hello > > + > > See test/py/README.md for more information about the pytest suite. > > > > +See :doc:`tests_sandbox` for how to run tests directly (not through > > pytest). > > + > > > > tbot > > ---- > > diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst > > new file mode 100644 > > index 00000000000..85bbd4f6734 > > --- /dev/null > > +++ b/doc/develop/tests_sandbox.rst > > @@ -0,0 +1,79 @@ > > +.. SPDX-License-Identifier: GPL-2.0+ > > + > > +Tests Under the Hood > > +==================== > > + > > +Running sandbox tests directly > > +------------------------------ > > + > > +Typically tests are run using the pytest suite. This is easy and always > > gets > > +things right. > > No, in the contrary: > > It is difficult to set up Python testing for real hardware. Running > non-Python tests from the console is trivial both on the sandbox and on > real hardware. OK I will reword it. > > > + > > +But it is also possible to run some sandbox tests directly. For example, > > this > > +runs the dm_test_gpio() test which you can find in test/dm/gpio.c:: > > + > > + $ ./u-boot -T -c "ut dm gpio" > This works only on the sandbox. To create examples assume that the shell > is already started: The whole point of this file is to describe using sandbox! > > => ut dm gpio > > and > > => bootefi selftest > > We should add a man-page for the ut command in /doc/usage/. OK, please go ahead. > > > + > > + > > + U-Boot 2021.01 > > + > > + Model: sandbox > > + DRAM: 128 MiB > > + WDT: Started with servicing (60s timeout) > > + MMC: mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD) > > + In: serial > > + Out: vidconsole > > + Err: vidconsole > > + Model: sandbox > > + SCSI: > > + Net: eth0: eth@10002000, eth5: eth@10003000, eth3: sbe5, eth6: > > eth@10004000 > > + Test: dm_test_gpio: gpio.c > > + Test: dm_test_gpio: gpio.c (flat tree) > > + Failures: 0 > > + > > Only the text below is sandbox specific. > > > +The -T option tells U-Boot to run with the 'test' devicetree (test.dts) > > instead > > %s/U-Boot/the U-Boot sandbox/g > > > +of -D which selects the normal sandbox.dts - this is necessary because many > > +tests rely on things in the test devicetree. If you try running tests > > without > > %s/things/nodes/ ? or properties > > > +-T then you may see failures, like:: > > + > > + $ ./u-boot -c "ut dm gpio" > > + > > + > > + U-Boot 2021.01 > > + > > + DRAM: 128 MiB > > + WDT: Not found! > > + MMC: > > + In: serial > > + Out: serial > > + Err: serial > > + SCSI: > > + Net: No ethernet found. > > + Please run with test device tree: > > + ./u-boot -d arch/sandbox/dts/test.dtb > > + Test: dm_test_gpio: gpio.c > > + test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, > > &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22) > > + Test: dm_test_gpio: gpio.c (flat tree) > > + test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, > > &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22) > > + Failures: 2 > > + > > +The message above should provide a hint if you forget. Even running with -D > > If I forget what? Please, complete the sentence. OK. Regards, Simon