To this point, our testing strategy uses Yocto, but is not contained within Yocto. As part of a build, we produce two images: (1) device image, and (2) image containing everything necessary to run tests against a device. The second part is important to us, as we want a separation between the device and tests run against the device. Tests against a device are always performed across the network, between (1) and (2). We want anyone to easily be able to run or develop tests without needing to setup a workstation or a workspace. A separate testing image, runnable in QEMU, allows anyone to pick up a build and start testing with minimal effort. We also build a toolchain containing QEMU to ensure that everyone uses the same version and features of QEMU. Finally, we have many tests, some which take longer than 24 hours to run. We want to parallelize these tests and run them across multiple systems at the same time. Because of this, we take the output from a build and run tests. Some tests may make sense to run every build, but other tests are run periodically throughout the day independent of the build.
Both our build system and test system are based on Jenkins. One job will build the code, and another will run tests. The test job is configured to run multiple types of tests in multiple configurations, all concurrently. As our tests are always run across the network, tests need not know if the device under test is a simulation or hardware. Jenkins is used to manage requests for hardware; if a configuration requires the use of a piece of hardware, that part of the job blocks until the hardware is available. When the hardware becomes available, Jenkins will allow that part of the job to start. As for the arrangement of the hardware, we have a testbed consisting of several devices. Each device is connected to the network and hooked up to a power switch that is connected to the network. When a test is run against a device, that device is reprogrammed, power cycled, then the tests are run. To answer some of your specific questions: * What hardware setup do you use for automated testing? > There is a testbed of devices, each connected to the network with a fixed IP address. Jenkins is used to run tests, and also manages access to hardware via labels. When a Jenkins test job requires a piece of hardware and that hardware is available, a configuration file is used to map the device to use to its IP address. From there, tests are run against the device. * Do you use any software to manage the hardware? Does this include any > existing open source tools? > The only software we used to manage the hardware is Jenkins. We do use other software to manipulate the hardware. For example, a power switch accessible over the network, a DHCP server to assign the proper address, etc. Likely this will not help you develop a hardware testing solution in Yocto. Maybe it will give you a data point though. - George
_______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto