Hi Teemu and Adam, @Teemu:
* Automated hardware based regression testing: The automated hardware regression testing using a logic analyzer would certainly be interesting. However, I think the first version of the automated hardware based test system will be much more simplistic (basically just boards connected to an internet connected computer). The main reason for this is that we (well I and Oleg) came to the conclusion that we cannot possible operate an automatic hardware based test system for each and every board we support (as you know the number of boards we support is rising steadily). So we had the idea to make the system distributed and easy to setup (both hardware- and software-wise). Ideally, such a system would then allow external entities (companies / universities etc.) to setup their own test system for the hardware they care about (or happen to have). The test coordination / building of binaries would then be done by a central server (which would be controlled by us). * Virtualization / Simulation / CI systems: I think no matter which CI system we will pick we can put it into Docker or something similar and we will definitely insure that it is hassle-free to use. * RF: I have to admit that I hadn't had the time to look into the specifics of your test system. Since, I really like the way you can specify test cases in RF I would like to hear more about your approach. It would be cool if Asif could join us and give us a quick technical run down of the system (BTW: It might not be that we will copy your testing approach but I am definitely intrigued by RF). @Adam: Testing the network stack is one of the major concerns right now. AFAIK Martine has put quite some effort into making the network stack testable (she also wrote a bunch of unit tests for network stack related modules). Concerning the code coverage: I think its safe to say that not a lot of modules are covered by actual unit tests. Most tests in the test directory are what I would call simple acceptance tests. As part of our effort to improve the testing system we will definitely try to convert some of those tests either into more sophisticated acceptance tests (using RF) or unit tests (or both if applicable / sensible). Regarding your offer: First of all thank you for your offer. The way I see it there is currently no need for a rush concerning the hardware. We (well mostly I and maybe Martine) will need to develop / configure our new test system first. So it could take a few weeks until we have something you could install on a linux box. Best, Philipp 2014-11-06 19:59 GMT+01:00 Adam Hunt <voxa...@gmail.com>: > I expect to have some real hardware in hand (CC2538 based boards of my own > design, and possibly some MSP430 boards after that) in the coming weeks > (months if I don't get into gear). I'd be more than willing to contribute > some whatever time is needed to running tests on real hardware if it's > helpful (assuming that I have the gear, my LA isn't the greatest but that > could change). > > On a related note, how much (if any) of RIOT's code base is covered by > real unit tests? Has any thought been given to a TDD type of unit test > battery? With the all the refactoring going on in the network stack this > might be the perfect time to get serious about writing some tests. > > On Wed, Nov 5, 2014 at 10:58 PM, Teemu Hakala <te...@iki.fi> wrote: > >> On 5.11.2014, at 17:42, Martine Lenders <authmille...@gmail.com> wrote: >> >> 2014-11-05 16:24 GMT+01:00 Philipp Rosenkranz < >> philipp.rosenkr...@fu-berlin.de>: >> >>> Dear RIOTers, >>> >>> in order to discuss possible enhancements to our current test >>> setup I would like to invite you to a dedicated RIOT testing meeting. >>> >>> The main topics to discuss will be: >>> >>> * Automated hardware based regression testing. >>> >> >> This is where we have existing code for Robot Framework. ELL-i is able to >> run unit tests >> in real hardware and apply tests with a logic analyser. >> >> >> * CI systems: Jenkins / buildbot or both? >>> * Simulations / Virtualization as part of our testing strategy. >>> >> >> I'd like that whatever test system it is, an individual developer could >> run it even offline. >> For this goal, virtualization seems to be the easiest path. >> >> >> We also should discuss robot framework [1] as suggested by ELL-i at the >> workshop. I did not look into it too much, but from as much as I gathered >> it might be helpful to automate a lot of already existing tests in our >> `tests/` directory. It's also integratable into Jenkins if this is >> important to you [2]. >> >> >> RF is Python and its test cases are in wiki markdown format. This makes >> it easy to separate test spec from the testing framework itself allows >> nonprogrammers to participate in some of the generating of test cases. I'm >> hoping Asif can join the meeting as he is the key RF person for ELL-i and >> has written bidirectional Python<->C hooks for running unit tests against >> the ELL-i emulator. The emulator itself is emulating a MCU including >> register controlled peripherals so we have some interesting tests coming >> up. It could be used in Riot OS context as well. >> >> - t >> >> >> _______________________________________________ >> devel mailing list >> devel@riot-os.org >> http://lists.riot-os.org/mailman/listinfo/devel >> >> > > _______________________________________________ > devel mailing list > devel@riot-os.org > http://lists.riot-os.org/mailman/listinfo/devel > >
_______________________________________________ devel mailing list devel@riot-os.org http://lists.riot-os.org/mailman/listinfo/devel
