Gentles. Last week we added a lot more tests for series/OSes. I can summarise what we setup in 2013, what we did in the first weeks of 2014, and what is next.
We stood up CI in October 2013 using scripts I wrote to manage and verify releases. The tests took the tip of the stable or devel branches and made a release candidate, then tested it against all the CPCs + local-provider (5 substrates). The functional tests tested many things at once and were brittle. We ran the tests often to discount intermittent failures. By the end of December, we had separated our tests into discrete parts We actually ran fewer tests each week because we saw fewer intermittent failures. We tested the tip of deploy and upgrade cases for devel and stable branches against the 5 substrates. CI can test any developer branch on demand, which was employed to verify possible fixes for the bugs that blocked 1.17.0. CI produces the release tarball. It also produces test debs. While we don't encourage anyone to used them, anyone who wants to test the bleeding edge can. January's focus has been on adding series/OSes dimensions to the tests. We started January only testing precise on amd64. We added unit tests first because we knew trusty wasn't a condition for merger. We also added precise because we wanted to test based solely on the juju-core dependencies.tsv (the juju-core lander requires human intervention to set the deps). The unit tests did not pass on trusty, but the devs were aware of this, and were already landing fixes. We added the trusty unit tests to the release group of tests last week when developers signalled. All pass, hurrah! We add Windows client building and testing last week. CI makes the juju installer with each release we test. Nate and I can now fall under a bus knowing the win-client will be built without us. The win-client deployment test fails however. The win-client rejects public IP on bootstrap and waits for the private IP to come available. We need this fixed of course to release 1.18.0. Last week, we also added hourly CPC health checks. We use the latest juju-core release to bootstrap and deploy services. This allows CI to know when there is a problem with a cloud, not with juju. We saw two cases last year where juju failed to work with Azure because of API and regions changes. CI caught the problems, but reported them as juju release candidate failures. We now report the issue a critical cloud problem that affects all juju users. We are adding trusty deployment and upgrade tests this week. This will add 10 more tests to the release group of tests that we run per release candidate. We are also tuning the rules of what to retest. We manually retest intermittent failures at this time, or re-running all the tests for a release candiate. Our next additions will be to add an architecture dimension and ecosystem tools like the gui and quickstart. We will see these in February. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
