On 12.06.23 17:58, Gedare Bloom wrote:
On Mon, Jun 12, 2023 at 12:56 AM Sebastian Huber
<sebastian.hu...@embedded-brains.de> wrote:
On 01.06.23 22:06, Gedare Bloom wrote:
On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom<ged...@rtems.org> wrote:
---------- Forwarded message ---------
From: Sebastian Huber<sebastian.hu...@embedded-brains.de>
Date: Wed, May 31, 2023 at 10:31 AM
The existing tests in the RTEMS test suite are basically BSP
independent. This patch set introduces BSP-specific validation tests.
These tests are disabled for other BSPs through the build system, for
example:
spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml
[...]
cxxflags: []
enabled-by: sparc/gr712rc
features: c cprogram
[...]
The use of the enabled-by: field to control the BSP-specific tests
looks reasonable. However, we should decide where/how any BSP-specific
tests should reside. It looks to me like the current approach is to
dump all test files in a single monolithic 'validation' directory, and
let the user/script sort it out. This results in a mix of tests
intended for all targets, and some for specific targets. This is
pretty much non-maintainable from my point-of-view without some
additional tool support. Correct me if I'm wrong.
Yes, this is the current approach. There is no strict one-to-one
relationship of test cases and test suites. The file names are somewhat
descriptive, for example:
ls -l *leon3*
-rw-r--r-- 1 sebastian_h domain users 3739 31. Mai 10:44
tc-bsp-sparc-leon3-gr712rc.c
-rw-r--r-- 1 sebastian_h domain users 6138 31. Mai 10:44
tc-fatal-bsp-sparc-leon3-shutdown.c
-rw-r--r-- 1 sebastian_h domain users 5135 31. Mai 10:44
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
-rw-r--r-- 1 sebastian_h domain users 2723 31. Mai 10:44
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.h
-rw-r--r-- 1 sebastian_h domain users 5406 31. Mai 10:44
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
-rw-r--r-- 1 sebastian_h domain users 2753 31. Mai 10:44
tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.h
-rw-r--r-- 1 sebastian_h domain users 5270 31. Mai 10:44
tr-fatal-bsp-sparc-leon3-clock-initialization.c
-rw-r--r-- 1 sebastian_h domain users 2681 31. Mai 10:44
tr-fatal-bsp-sparc-leon3-clock-initialization.h
-rw-r--r-- 1 sebastian_h domain users 2488 31. Mai 10:44
ts-bsp-sparc-leon3-gr712rc.c
-rw-r--r-- 1 sebastian_h domain users 2845 31. Mai 10:44
ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
-rw-r--r-- 1 sebastian_h domain users 2919 31. Mai 10:44
ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
-rw-r--r-- 1 sebastian_h domain users 2797 31. Mai 10:44
ts-fatal-bsp-sparc-leon3-clock-initialization.c
-rw-r--r-- 1 sebastian_h domain users 3148 31. Mai 10:45
ts-fatal-bsp-sparc-leon3-shutdown-response.c
-rw-r--r-- 1 sebastian_h domain users 4909 31. Mai 10:44
ts-fatal-bsp-sparc-leon3-shutdown.c
We could also introduce subdirectories to organize things. The test
framework prints out the file name in messages, so it would be nice if
they remain unique. With subdirectories this would lead to longer path
names, for example
testsuites/validation/sparc/leon3/tc-bsp-sparc-leon3-gr712rc.c
I see. Maybe it makes sense to have all BSP-specific tests in a bsps
subdirectory, with the unique names encoded to ensure the arch/bsp
combination appears in the filename, such as:
testsuites/validation/bsps/tc-bsp-sparc-leon3-gr712rc.c
If so, 'bsp-' can probably be omitted from the filename. This way,
architecture-specific testing may also be easily possible, like
tc-sparc-something.c
This is a nicely balanced and easy approach. Place BSP-specific tests in
a "bsps" directory. Place architecture-specific tests in a "cpu" directory.
I would like to discuss possible ways to manage the integration of
tests that are conditionally-built based on the arch/bsp tuple. We
should have clear guidance for others who want to add such tests in
the future, or who would modify existing tests.
Yes, this makes sense. We could add a new section for BSP-specific tests to:
https://docs.rtems.org/branches/master/eng/req/howto.html
For a pre-qualified BSP you have to specify the fatal errors and write
validation tests for it. Other BSP-specific specification and validation
may be necessary for the kernel IO device driver, cache controller
support, memory management unit initialization, memory protection unit
initialization, etc.
Good, that would definitely be important to document.
Ok, I will add something to this section.
I think we should have some of the BSP-specific tests under some other
location in testsuites, while others would be under validation where
they are used for pre-qualification?
Maybe it is sensible to introduce testsuites/bsps/ also.
For the unit tests I would use testsuites/unit/bsps and testsuites/unit/cpu.
I'm not entirely clear about the difference between 'validation' and
any other kind of tests, such as where specific tests should be
located.
The validation tests are linked to a specification item.
Even for normal BSPs, some unit tests would be helpful. For example, I
had to fix _AArch32_PMSA_Initialize() twice:
https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c
Unit tests would have probably found the errors before applications hit
the special cases.
Something like testsuites/bsps/arm-aarch32-psma-init.c (or whatever
file naming convention we like to decide on. I forget the meaning of
tc/tr/ts/tx but recall they are about specific kinds of testing for
validation.)
tc - Test case
ts - Test suite
tr - Test runner
tx - Test extensions (support code)
Gedare
--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.hu...@embedded-brains.de
phone: +49-89-18 94 741 - 16
fax: +49-89-18 94 741 - 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.hu...@embedded-brains.de
phone: +49-89-18 94 741 - 16
fax: +49-89-18 94 741 - 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
