Hello! Thanks for the effort on testing NuttX! Testing is what we need to have a safe, trustworthy OS. That's what allows us to keep growing with confidence that a documentation change did not brake the scheduler.
1. I personally don't like the granularity, and would keep to a minimum given our testing capabilities. So I would say: - Base tests are what we need and we should focus on this. - Extended tests seem very specific and do not fit CI: it is its own category. - Specific tests require more infrastructure. I think this should be the focus after the base tests. - Custom tests: we currently use Pytest for everything, but will keep tracking the progress of NTFC and also we are open to sharing our tests, which run on top of defconfigs. 2. Base tests should be mandatory with the test cases mentioned. I would add "ps" command test (this gives more headaches than you would think) and a quick file system test. We should have the execution time of those tests logged. 4. I don't like it. Testing many things together does not mean they also work individually, there are many corner cases that appear when we start to increase the features available in a single config. 5. Not really, see my answer above. For us I believe CI should be granular, just as unit tests would be in different applications. Also by doing this, you would not have the same baseline of tests because the grouped config may be too big for some devices. There would be not direct comparison. 6. See 5. Thank you again, and happy new year all. Filipe ________________________________ From: Tomek CEDRO <[email protected]> Sent: Thursday, January 1, 2026 4:20 PM To: [email protected] <[email protected]> Subject: [DISCUSSION] board configurations vs testing [External: This email originated outside Espressif] Hello world :-) As there are some works ongoing to improve NuttX testing (nxdart, ntfc, nuts) we are coming back to to discussion about existing (and possibly new) board configurations that would help in testing. Initial discussion [1] proposed four levels of testing: base, extended, specific, custom. Base should cover all boards and constitute mandatory implementation. Extended tests can be optional and contain stresstests and benchmarks. So far lots of boards had separate nsh, ostest, coremark, getprime, configurations, that would fit in as the candidates for base and extended tests. Looks like all boards hould have base and extended tests defined. But we did not define them back then. After recent release testing I noted down missing ostest for some boards [2] to keep things aligned to what we already have. Raiden raised concern that this makes NuttX more complex and came out with idea to bundle all testing utilities where possible on bigger boards into one configuration - because that would simplify citesting, require only one build, and allow different runtime tests to be performed with a single configuration. I like the idea myself.. with caution that not all boards would fit such a bundle :-) For now configurations are a bit incoherent - for instance some :nsh configurations only contain bare nsh, with different built-in commands, some additionally contain ostest, getprime, etc, while other boards have them as separate configurations. The main idea is to clean up existing configurations that would keep them coherent, predictable, and ci/automation friendly. There are several things to discuss to get things coherent and documented: 1. How to organize test configurations [1]? Is the 4 tier granularity okay? Do we want something else? 2. What should we consider as mandatory :test-base configuration? What should it contain? nsh? free? ostest? coremark? getprime? What exact applications and script? 3. What should we consider as optional :test-ext configuration? What should it contain? coremark? getprime? What exact application and script? 4. Do we agree to have one big test configuration for big boards where they fit? 5. If we agree on (4) then would is be okay to remove existing configurations with applications that would be part of the :test-base and :test-ext bundle (i.e. :ostest that would go to :test-base and :coremark that would go to :test-ext)? This would simplify CI with less configurations to build. 6. What happens if smaller board cannot fit all of the test-base applications? Should we still have separate configurations for example test-base-ostest / old ostest, test-ext-getprime / old getprime, etc? 7. What should be the name for the test bundle? Do we want to keep it under "nsh" or new "test-base"? Note from Raiden suggests that less configurations is better for CI/build/automation. I know this discussion may result in existing configurations change, thus may be breaking change, thus we may need a vote after details are set if necessary :-) If there is anything missing please add! Any hints welcome :-) Tomek [1] https://github.com/apache/nuttx/issues/15773 [2] https://github.com/apache/nuttx/issues/17721 -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
