There are going to be some changes for Yocto Project QA in the 2.7 cycle. Intel is planning to scale back its involvement and wants to transition it to the wider community.
Unfortunately we don't have people we can transition this to so a group of us have been analysing what the implications of this are and how we can best mitigate the changes. The short summary is don't panic. We will lose test coverage in some areas but we should be able to keep the core of the project strong and to the quality we're used to. For more info, read on. The first part of this exercise was to figure out what is being tested by the autobuilder and what is being tested by the QA teams today. The autobuilder piece is quite clear, it tests the configurations summarised by this config.json file: http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder-helper/tree/config.json Intel has slides showing what they test, I'm primarily interested in what the delta is between what gets tested by QA (manually or automatically) compared with the autobuilder. My summary of the delta in testing is: * Real hardware targets (Intel tests genericx86*, WindRiver tests beaglebone-yocto, mpc8315e-rdb and edgerouter) * manual OE-Core tests * oe-selftest on each supported distro * the other package manager backends (ipk + deb) * toaster * CROPS * eclipse-plugin * trigger runs of builder performance data for release builds We also have the complexity that the QA process has traditionally been tightly integrated into testopia (a plugin within bugzilla). We really need to drop testopia, its unmaintained holding us on an elderly bugzilla version. Testopia has two major functions, it contained all our testcases and handled the test execution, results handling and reporting. As a first step in moving forward I therefore asked the existing tests all be documented in lib/oeqa within OE-Core. Those patches landed late in 2.6, in particular with the population of the "manual" directory: http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/manual This now means OE-Core is definitive for the test cases we have (when looked at with the other selftest/runtime/sdk tests) and means we no longer need testopia for test case storage/management. For results tracking, I'm asking that builds that run tests generate a json results file containing the test results. This code merged late in 2.6: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=fc16be17878486181d460416618f4f06d2be40e3 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=e73bcdd0590fbc82da515b37e5cf32726f231cf9 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d89e06083eeea1bebdaa64f809e7e0a328e282f5 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=4ef72d556f706e301e5155f5f228c38431acafe6 I also added code which connects up ptest results into the main results file: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=fcf55f58ae73b4e54ea853c816c0fef0f33ca46c What we don't have yet is results management, the results aren't collected from the autobuilder, or analysed. There are patches from Ee Peng with some proposed code to do this on the mailing list: http://lists.openembedded.org/pipermail/openembedded-core/2018-November/157403.html but this isn't ready for merging yet. Going back to the above list and taking each item in turn: * Real hardware targets - Intel and Wind River are going to continue testing these * Manual OE-Core tests - these are now documented in OE-Core. We need to go through and analyse and clean the ones which don't make sense or we don't need.and In order to run these going forward they will need to be automated and likely won't be run until this is done. * oe-selftest on each supported distro - the public autobuilder should be able to be configured to do this (currently it runs randomly on one distro). * the other package manager backends (ipk + deb) - there was a bug in the public autobuilder, this was fixed and is now being tested automatically. * toaster - David Reyna is looking into selenium automation but likely won't be tested without automation * CROPS - no coverage plan * eclipse-plugin - no coverage plan This means there are a few things we need to work on. I've therefore created this rough action list of things we need to do: * Review the manual test cases and remove any which are obsolete or unneeded * Create a list of manual test cases which are important and need to be automated (file an enhancement bug for each?) * Add ptest running targets to the autobuilder to start collecting ptest results for arm and x86 machines under qemu. * Add functionality to the autobuilder to enable oe-selftest to run once per distro * Handle automated results collection on the autobuilder * Create/merge results aggregation and analysis tool * Run results aggregation and analysis on the autobuilder * Handle triggering and collection of build performance data for release builds * Upgrade bugzilla to a new version and drop testopia (keep around an old instance for archive purposes?) * How to run our test suites for boards like beaglebone remotely on a LAVA board farm? * Can we create some mechanism to allow easier end developer testing on real hardware ('plug a board into their laptop and go')? * Separate logic on autobuilder for a lighter test run and a longer more comprehensive one The above should all really become bugzilla items and I'd welcome help with working on them. I know some people are already looking at some of the items (e.g. Anibal on the LAVA pieces). There is also a complication in all this which is stable releases. Somehow we need to not just automate master but also automate our point release testing for the stable series. This was why it was important to get automated results collecting into 2.6. I'm thinking about this and some pieces like the autobuilder enhancements should work for older releases too. We may need to backport some "features" against our normal policy to ensure we continue to have good testing for them. The email is long enough. I'm sure there are pieces I've forgotten but I'll stop here and ask for comments, concerns and other feedback. Bottom line is, we shouldn't panic but we are going to need changes and unless people step up, some things will fall through cracks (eclipse plugin and crops in particular). I'm very open to help with any of the above. Cheers, Richard _______________________________________________ Openembedded-architecture mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
