On 06/18/2012 01:43 PM, Richard Purdie wrote:
On Thu, 2012-06-14 at 15:26 +0200, Björn Stenberg wrote:
Many source packages include their own package test suites. We are
looking at running these tests on target in a structured way, and I
would like a discussion about how to best fit this into the Yocto
framework.
The actions that need to be performed for a package test are roughly:
1) Build the test suite
2) Make the test suite appear on target
3) Run the test suite
4) Parse the results
Each action can be done in several ways, and there are different
considerations for each solution.
1) Build the test suite
-----------------------
Many package tests are simply bash scripts that run the packaged
binaries in various ways, but often the test suites also include
binary tools that are not part of the normal package build. These
tools have to be built for package testing to work.
Additionally, many packages build and run the test in a single
command, such as "make check", which is obviously unsuitable for
cross-compiled packages.
We can solve this in different ways:
a) Run the test build+run jobs on target. This avoids the need to
modify packages, but building code on target can get quite expensive
in terms of disk space. This in turn means many tests would require a
harddisk or network disk to run.
b) Patch the makefiles to split test building and test running.
Patching makefiles mean we get an additional maintenance and/or
upstreaming burden, but we should be able to do this in ways that are
acceptable to upstream. This is our suggestion.
There are some good questions here :)
Yes there are!
I think in general, we should be aiming for b) since that falls well
into some of the ideas below. Upstreams should be amenable to splitting
these into two targets (assuming "make check" just runs one after the
other) so we shouldn't have to carry patches in most cases long term.
I think you both might be missing some cases here. The problem is that
'make check' does not have a standardized meaning.
From the top of my head, I can think of these different variants:
1) Function test level with testing run on target, package cross-compiled
2) Function test level with testing run on host, package cross-compiled
3) Function test level with testing run on host, package compiled for host
4) Unit test level with testing run on target, package cross-compiled
5) Unit test level with testing run on host, package compiled for host
I think that we need to be able to separate the different uses of 'make
check' to be able to support testing nicely.
2) Make the test suite appear on target
---------------------------------------
The test suite and utilities obviously have to be executable by the
target system in order to run. There are a few options for this:
a) Copy all test files to the target at test run time, from the build
dir, using whatever means available (scp, ftp etc). This limits
testing to targets/images with easy automatic file transfer abilities
installed.
b) NFS mount the build dir to access the full work dir and hence the
test code. this limits testing to targets (and images) with network
+nfs support. Plus it blends the build env and runtime env in an
unpleasant way.
c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that
contain all test files and add those to the image and hence the
rootfs. This is our suggestion.
I like the idea of test packages.
Would be very nice for functional tests, but it will not be so simple
for unit testing. For unit tests I belive you need some build target
too, which leads back to the comment above.
It might be a lot of autoconf hacking to get this working, don't know if
there are any widely accepted standard targets for it.
/Regards
Fredrik Hugosson
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
