On Tue, Jun 13, 2023 at 8:56 AM Sebastian Huber <sebastian.hu...@embedded-brains.de> wrote: > > > > On 12.06.23 17:58, Gedare Bloom wrote: > > On Mon, Jun 12, 2023 at 12:56 AM Sebastian Huber > > <sebastian.hu...@embedded-brains.de> wrote: > >> > >> > >> > >> On 01.06.23 22:06, Gedare Bloom wrote: > >>> On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom<ged...@rtems.org> wrote: > >>>> ---------- Forwarded message --------- > >>>> From: Sebastian Huber<sebastian.hu...@embedded-brains.de> > >>>> Date: Wed, May 31, 2023 at 10:31 AM > >>>> > >>>> The existing tests in the RTEMS test suite are basically BSP > >>>> independent. This patch set introduces BSP-specific validation tests. > >>>> These tests are disabled for other BSPs through the build system, for > >>>> example: > >>>> > >>>> spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml > >>>> [...] > >>>> cxxflags: [] > >>>> enabled-by: sparc/gr712rc > >>>> features: c cprogram > >>>> [...] > >>> The use of the enabled-by: field to control the BSP-specific tests > >>> looks reasonable. However, we should decide where/how any BSP-specific > >>> tests should reside. It looks to me like the current approach is to > >>> dump all test files in a single monolithic 'validation' directory, and > >>> let the user/script sort it out. This results in a mix of tests > >>> intended for all targets, and some for specific targets. This is > >>> pretty much non-maintainable from my point-of-view without some > >>> additional tool support. Correct me if I'm wrong. > >> > >> Yes, this is the current approach. There is no strict one-to-one > >> relationship of test cases and test suites. The file names are somewhat > >> descriptive, for example: > >> > >> ls -l *leon3* > >> -rw-r--r-- 1 sebastian_h domain users 3739 31. Mai 10:44 > >> tc-bsp-sparc-leon3-gr712rc.c > >> -rw-r--r-- 1 sebastian_h domain users 6138 31. Mai 10:44 > >> tc-fatal-bsp-sparc-leon3-shutdown.c > >> -rw-r--r-- 1 sebastian_h domain users 5135 31. Mai 10:44 > >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c > >> -rw-r--r-- 1 sebastian_h domain users 2723 31. Mai 10:44 > >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.h > >> -rw-r--r-- 1 sebastian_h domain users 5406 31. Mai 10:44 > >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c > >> -rw-r--r-- 1 sebastian_h domain users 2753 31. Mai 10:44 > >> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.h > >> -rw-r--r-- 1 sebastian_h domain users 5270 31. Mai 10:44 > >> tr-fatal-bsp-sparc-leon3-clock-initialization.c > >> -rw-r--r-- 1 sebastian_h domain users 2681 31. Mai 10:44 > >> tr-fatal-bsp-sparc-leon3-clock-initialization.h > >> -rw-r--r-- 1 sebastian_h domain users 2488 31. Mai 10:44 > >> ts-bsp-sparc-leon3-gr712rc.c > >> -rw-r--r-- 1 sebastian_h domain users 2845 31. Mai 10:44 > >> ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c > >> -rw-r--r-- 1 sebastian_h domain users 2919 31. Mai 10:44 > >> ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c > >> -rw-r--r-- 1 sebastian_h domain users 2797 31. Mai 10:44 > >> ts-fatal-bsp-sparc-leon3-clock-initialization.c > >> -rw-r--r-- 1 sebastian_h domain users 3148 31. Mai 10:45 > >> ts-fatal-bsp-sparc-leon3-shutdown-response.c > >> -rw-r--r-- 1 sebastian_h domain users 4909 31. Mai 10:44 > >> ts-fatal-bsp-sparc-leon3-shutdown.c > >> > >> We could also introduce subdirectories to organize things. The test > >> framework prints out the file name in messages, so it would be nice if > >> they remain unique. With subdirectories this would lead to longer path > >> names, for example > >> > >> testsuites/validation/sparc/leon3/tc-bsp-sparc-leon3-gr712rc.c > >> > > I see. Maybe it makes sense to have all BSP-specific tests in a bsps > > subdirectory, with the unique names encoded to ensure the arch/bsp > > combination appears in the filename, such as: > > testsuites/validation/bsps/tc-bsp-sparc-leon3-gr712rc.c > > If so, 'bsp-' can probably be omitted from the filename. This way, > > architecture-specific testing may also be easily possible, like > > tc-sparc-something.c > > This is a nicely balanced and easy approach. Place BSP-specific tests in > a "bsps" directory. Place architecture-specific tests in a "cpu" directory. > > > > >>> > >>> I would like to discuss possible ways to manage the integration of > >>> tests that are conditionally-built based on the arch/bsp tuple. We > >>> should have clear guidance for others who want to add such tests in > >>> the future, or who would modify existing tests. > >> > >> Yes, this makes sense. We could add a new section for BSP-specific tests > >> to: > >> > >> https://docs.rtems.org/branches/master/eng/req/howto.html > >> > >> For a pre-qualified BSP you have to specify the fatal errors and write > >> validation tests for it. Other BSP-specific specification and validation > >> may be necessary for the kernel IO device driver, cache controller > >> support, memory management unit initialization, memory protection unit > >> initialization, etc. > >> > > Good, that would definitely be important to document. > > Ok, I will add something to this section. > > > > > I think we should have some of the BSP-specific tests under some other > > location in testsuites, while others would be under validation where > > they are used for pre-qualification? > > > > Maybe it is sensible to introduce testsuites/bsps/ also. > > For the unit tests I would use testsuites/unit/bsps and testsuites/unit/cpu. > I like this.
> > > > I'm not entirely clear about the difference between 'validation' and > > any other kind of tests, such as where specific tests should be > > located. > > The validation tests are linked to a specification item. > Ok > > > >> Even for normal BSPs, some unit tests would be helpful. For example, I > >> had to fix _AArch32_PMSA_Initialize() twice: > >> > >> https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c > >> > >> Unit tests would have probably found the errors before applications hit > >> the special cases. > >> > > Something like testsuites/bsps/arm-aarch32-psma-init.c (or whatever > > file naming convention we like to decide on. I forget the meaning of > > tc/tr/ts/tx but recall they are about specific kinds of testing for > > validation.) > > tc - Test case > > ts - Test suite > > tr - Test runner > > tx - Test extensions (support code) > Thanks Gedare > > > > Gedare > > > >> -- > >> embedded brains GmbH > >> Herr Sebastian HUBER > >> Dornierstr. 4 > >> 82178 Puchheim > >> Germany > >> email: sebastian.hu...@embedded-brains.de > >> phone: +49-89-18 94 741 - 16 > >> fax: +49-89-18 94 741 - 08 > >> > >> Registergericht: Amtsgericht München > >> Registernummer: HRB 157899 > >> Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler > >> Unsere Datenschutzerklärung finden Sie hier: > >> https://embedded-brains.de/datenschutzerklaerung/ > > -- > embedded brains GmbH > Herr Sebastian HUBER > Dornierstr. 4 > 82178 Puchheim > Germany > email: sebastian.hu...@embedded-brains.de > phone: +49-89-18 94 741 - 16 > fax: +49-89-18 94 741 - 08 > > Registergericht: Amtsgericht München > Registernummer: HRB 157899 > Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler > Unsere Datenschutzerklärung finden Sie hier: > https://embedded-brains.de/datenschutzerklaerung/ _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel