Hello all, I recently set up some Calxeda ARM nodes that we got for QA and improved autopkgtest enough so that we can efficiently and robustly run autopkgtests using the LXC runner [1]. As a first smoketest I run all our 310 existing autopkgtests except linux and libreoffice on them.
I wanted to see whether the four servers, LXC, and autopkgtest are robust enough, and indeed they survived that. That's a major improvement from our first attempt to run them on Panda boards, which fell over more often than not. I set that off yesterday evening, and by today noon the last test has finished. Two of the nodes were done this morning already, but two others had the bad luck to catch tests which hung indefinitely and thus only timed out after 4 hours. The full raw logs are at [2]. I went through all failures of packages which don't fail in our production run with Qemu/KVM [3] and categorized/summarized them, notes are at [4]. Summary ------- * 48 tests out of the 310 tests failed which don't fail with KVM on x86. * 34 failures happen on amd64 with LXC as well, with the same error. I. e. they don't fail because of ARM, but because the test only runs in a container and not in a full VM. (Note, we don't currently have a kvm-like qemu on ARM, so we have to use LXC there) * 14 failures are ARM specific, i. e. they succeed on x86/LXC. * 2 packages fail which also fail on x86/LXC, but they have additional errors. So only 16 packages would need fixing to get their autopkgtests up to par with x86. That's a lot less than I initially feared. CI integration -------------- We want to put this into production soon. To avoid blocking on LXC or ARM/LXC specific failures, we will teach britney to only consider an autopkgtest a failure if that test ever succeeded. In fact we should do this for all architectures, as unfortunately 80% of the new autopkgtests that we get fail (they hardly ever get tested in a proper isolated environment by uploaders unfortunately). Fixing tests ------------ In some cases tests that work in qemu fail in LXC because the test tries to do kernel-level things like loading modules, accessing cgroups, etc. These tests just can't run in LXC, and there is little that we can do about it. We'll just continue running them in Qemu on x86 and don't run them on ARM. But in quite a lot of cases this is because the autopkgtest LXC runner is stricter than the "null" runner that we are currently using; for example, if your test has "Restrictions: build-needed" then in Qemu your test still has all the build dependencies installed, but the LXC runner resets the testbed between build and test; so your debian/tests/control needs to specify all runtime Depends:. (Note that you can use @builddeps@ in cases where this is appropriate) Fixing tests to work in LXC is greatly appreciated, so if you want your package to always work on ARM, and want dependencies that break your package be held in -proposed instead of landing and breaking your package, please make sure that you have a working LXC autopkgtest. Locally running tests in LXC ---------------------------- Dependencies: $ sudo apt-get install lxc autopkgtest Create an LXC container: $ sudo lxc-create -t ubuntu-cloud -n trusty-cloud -- -r trusty (You can use any other name with -n, of course) Run tests from package "pkgname" in the archive: $ sudo adt-run pkgname --- adt-virt-lxc --ephemeral trusty-cloud Run locally fixed tests for package "pkgname" with the packages from the archive: $ cd pkgname-1.1/ $ # fix debian/tests/ $ sudo adt-run -B --unbuilt-tree=. --- adt-virt-lxc --ephemeral trusty-cloud The difference between --built-tree and --unbuilt-tree only matters if your test has "Restrictions: build-needed"; then, use whichever state your tree is in. Build locally fixed package, install its built debs for the test, and run fixed tests from local source tree: $ cd pkgname-1.1/ $ # fix whatever $ sudo adt-run --unbuilt-tree=. --- adt-virt-lxc --ephemeral trusty-cloud Note that you can use the "-o /tmp/output-dir" option to adt-run if you want to put the autopkgtest logs into a directory. Let me know if you have any question. Thanks, and happy weekend! Martin [1] https://plus.google.com/u/0/107564545827215425270/posts/HktYxb8XdMy [2] http://people.canonical.com/~pitti/tmp/autopkgtest-arm/ [3] https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/ [4] http://people.canonical.com/~pitti/tmp/autopkgtest-arm/analysis.txt -- Martin Pitt | http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
signature.asc
Description: Digital signature
-- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel