On Tue, 2013-08-27 at 12:11 +0100, Paul Eggleton wrote: > Hi Darren, > > On Friday 23 August 2013 20:52:41 Darren Hart wrote: > > On Tue, 2013-08-13 at 12:00 +0100, Paul Eggleton wrote: > > > Automated testing on real hardware is something that a lot of us do > > > already, but the likelihood is that we've all been coming up with our own > > > solutions for this, so it would be really great if we could have a single > > > working solution to cover the generally applicable use cases. As a start > > > though it would be interesting to hear from people who have existing test > > > setups or who are generally interested in this area. I'd like to hear > > > feedback on the following: > > > * What hardware setup do you use for automated testing? > > > > I am starting to put something together using a customized 19" rack > > along the lines of those used by OSADL for their real-time farm: > > > > https://www.osadl.org/Test-Rack.test-rack.0.html > > > > This is a poor man's BladeCenter for embedded boards in that it provides > > a common interface to each board for the rack infrastructure. The board > > specific stuff sits on the trays themselves. The trays are easily > > removable to take a board to a desk for hands-on work. > > > > A smart-switch, web-power strip, and serial concentrator feed into a > > "gateway server" where services like conmux glue it all together. > > All makes sense. > > > > * Do you use any software to manage the hardware? Does this include any > > > existing open source tools? > > > > conmux > > dnsmasq > > udev (for static usb uart naming) > > > > The testrack has some early-stage tools to support it, but I'm still > > sorting those out and they aren't particularly robust for initial setup > > quite yet. > > Having a standard way to set up the software side for this kind of > infrastructure would certainly be worthwhile I think. One tricky part would > perhaps be supporting variations of infrastructure hardware; doing some > searches I haven't found a completely definitive open source project for > controlling remotely accessible power strips for example, although there are > a > few scattered projects for doing that with varying levels of recent > maintenance. > > > > * What would you like to see in a common framework to enable this kind of > > > testing? > > > > To simplify the board management you allude to above, I believe a pull > > model, rather than a push, is a better architecture. > > > > Your builders can complete builds and inject test records into a > > database with a priority. The boards sit in a > > boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and > > push the results back to the database. They select tests based on > > priority, and the builder doesn't have to do anything at all to manage > > the boards. This also decouples the builder from the test > > infrastructure. > > Interesting. This would present some challenges with the way the automated > tests are currently defined; there's nothing unsolvable that precludes them > being run outside of the build environment but they will have to be run > somewhere that has python and any required modules installed, presumably an > intermediate machine rather than the target machine itself (the latter would > not be impossible; but if we did do it that way you couldn't test an image > that didn't have python in it.)
A new test-image will need to be created regardless, I think including python is reasonable for the general case. For things like tiny, I suspect we would have a customized test-suite anyway, possibly written in C. > > We'd also need to come up with a standard definition for the results database > and tools to generate reports out of it, but that's something that will > likely > be needed regardless of the model used for running tests, especially when it > comes to recording the results from ptest. Agreed. -- Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
