On Mon, Mar 21, 2016 at 10:19:47AM -0400, Emilien Macchi wrote: > On Mon, Mar 21, 2016 at 9:59 AM, Steven Hardy <sha...@redhat.com> wrote: > > On Mon, Mar 21, 2016 at 09:41:42AM -0400, Emilien Macchi wrote: > >> On Mon, Mar 21, 2016 at 6:57 AM, Steven Hardy <sha...@redhat.com> wrote: > >> > On Fri, Mar 18, 2016 at 01:27:33PM +0000, arkady_kanev...@dell.com wrote: > >> >> Emilien, > >> >> > >> >> Agree on the rant. But not clear on concrete proposal to fix it. > >> >> > >> >> Spend more time “fixing” CI and use Tempest as a gate is a bit wage. > >> >> > >> >> Unless we test known working version of each project in TripleO CI > >> >> you are > >> >> dependent on health of other components. > >> > > >> > I've so far resisted replying to this thread, because while valid, many > >> > of > >> > the concerns expressed by Emilien are quite general complaints, and it's > >> > hard to reply with specific solutions. > >> > > >> > However work *is* going on to improve many of these problems, let's see > >> > if > >> > I can provide a summary, to clarify the various "concrete proposals" > >> > which > >> > do exist. > >> > > >> > 1. Core team & review velocity > >> > > >> > We've had a small and very overloaded core team for a while now, and this > >> > will be helped by expanding our community to include those who've been > >> > regularly contributing excellent work and reviews as core reviewers: > >> > > >> > http://lists.openstack.org/pipermail/openstack-dev/2016-February/087774.html > >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089235.html > >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089912.html > >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089913.html > >> > > >> > Note that I personally think it's absolutely fine for folks to be more > >> > expert in some subsystem and to focus review extra attention on e.g API, > >> > UI, Puppet or whatever. This subsystem-core model has been well proven > >> > in > >> > other projects, and folks will naturally broaden their areas of deeper > >> > knowledge over time. > >> > > >> > Related to this is movement of code, such as the puppet-tripleo > >> > refactoring > >> > mentioned by Michael - this has already started, and will help with > >> > providing a cleaner interface between the puppet and heat pieces (which > >> > will also help focus reviewer attention appropriately). > >> > >> Indeed, Michael, Dan & I are working on moving out the Puppet code > >> from THT to puppet-tripleo. > >> That's a nice move, and I appreciate TripleO team support on it. > >> > >> > 2. Day 1 developer experience > >> > > >> > This is closely related to the CI failure rate - there are efforts to > >> > integrate with the RDO tripleo-quickstart tooling, which simplifies the > >> > initial undercloud setup, and potentially makes consuming pre-built, > >> > validated undercloud images (probably output artefacts from our new > >> > periodic CI job) much easier. > >> > > >> > So, this will mean that both developers and CI can potentially be less > >> > regularly impacted by trunk regressions which often cause CI to fail, and > >> > break developer environments. > >> > > >> > https://review.openstack.org/#/c/276810/5 > >> > > >> > 3. CI coverage and trunk failure rate > >> > > >> > We've been working really hard to improve things here, which are really > >> > several inter-related issues: > >> > > >> > - Lack of Hardware capacity in the tripleo CI cloud > >> > - Frequent trunk regressions breaking our CI > >> > - Lack of coverage of some key features (network isolation, SSL, IPv6, > >> > upgrades) > >> > - Lack of coverage for vendor plugin templates/puppet code > >> > > >> > There's work ongoing to improve this from multiple perspectives: > >> > > >> > New periodic CI job (to be used for automated promotion of the > >> > current-tripleo repo, and for pre-built undercloud images): > >> > https://review.openstack.org/#/c/271370/ > >> > > >> > Add network isolation support to CI: > >> > https://review.openstack.org/#/c/288163/ > >> > > >> > Test SSL enabled in overcloud: > >> > https://review.openstack.org/#/c/281988/ > >> > > >> > CI coverage of IPv6: > >> > https://review.openstack.org/#/c/289445/ > >> > > >> > Discussion around better documented integration for third-party CI: > >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/088972.html > >> > >> Do we have plans to execute Tempest? > > > > This is something which has been discussed several times, but right now we > > don't have the time available to run it per-commit because we'll hit the > > job timeout. > > > > This situation will improve as we gain time e.g through use of cached > > pre-built images, but right now I think we could look at enabling it only > > on the periodic job when that is fully proven. > > > > Having said that, I should point out that tempest doesn't get us great > > coverage of some newer projects - e.g all Heat scenario coverage was moved > > out of the tempest tree, and other projects have done similar AFAIK, so we > > may end up with very sparse API surface tests (or nothing at all) in these > > cases. > > In Puppet OpenStack CI, we execute smoke tests (a few tests of each > service, and some scenarios), and some tests not in smoke, for Ironic, > Aodh, Horizon. > It takes ~15 minutes to execute them.
Thanks, useful datapoint - this kind of time budget may become possible after we've optimized things like the undercloud image building. > This is how we proceed: > https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L116-L134 > > I'm asking because to me Tempest is the official project for OpenStack > testing and if we enable third party CI later, with external CI, we > need to find a common way to test TripleO clouds. I guess this is the point of my earlier comment - despite being "the" official project for OpenStack testing, tempest has very deep testing of certain projects (e.g Nova) and very shallow (API only) testing for others. So, while it's probably a good start, it's not a single solution unless you can configure things to also run per-project functional/integration tests. Steve __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev