On Mon, Aug 8, 2016 at 11:47 AM, James Slagle <[email protected]> wrote: > On Sat, Aug 6, 2016 at 8:17 PM, Paul Belanger <[email protected]> wrote: >> Greetings, >> >> 5 months ago fungi posted: >> >> [tripleo] becoming third party CI (was: enabling third party CI)[1] >> >> About having the discussion whether the existing TripleO CI should itself >> follow >> our third-party integration model instead of the current implementation >> relying >> on our main community Zuul/Nodepool/Jenkins servers. >> >> The result of the thread had some pros and cons, which I encourge people to >> re-read. >> >> At the Austin summit we continued the topic of moving tripleo-ci into 3rd >> party >> CI. Again, consensus could not be reached however we made some progress. I >> would take on the responsibility to help bring tripleo-test-cloud-rh1 more >> inline with openstack-infra tooling. >> >> That includes, but is not limited to: >> >> - Initial support for centos-7 jenkins slave (tripleo-ci) >> https://review.openstack.org/#/c/312725/ >> - Add centos-7 to tripleo cloud (project-config) >> https://review.openstack.org/#/c/311721/ >> - Revert "Revert "Migrate tripleo to centos-7"" (project-config) >> https://review.openstack.org/#/c/327425/ >> - Revert "Disable tripleo-test-cloud-rh1 until we have AFS mirrors" >> (project-config) >> https://review.openstack.org/#/c/349659/ >> - Add tripleo-test-cloud grafana dashboard >> https://review.openstack.org/#/c/351251/ >> >> And various other reviews adding AFS mirrors for centos / epel. Updates to >> tripleo-ci using our openstack-infra AFS mirrors along with providing general >> support for both tripleo-test-cloud-rh1 and tripleo-test-cloud-rh2. >> >> In a short amount of time, we've made great progress with >> tripleo-test-cloud-rh1, helping bring it more inline with openstack-infra >> tooling. While we are not finished, there is still some private >> infrastrucuture >> that tripleo-ci is depending on. I am confident in the next 3 months we >> should >> have that all replaced and using openstack community infrastruture. >> >> However on Friday[2], we started talking about tripleo-test-cloud-rh1 again >> in >> #openstack-infra and found ourselves revisiting the original email. It is all >> driven from the current effort from tripleo to start using move community >> clouds >> for running tripleo-ci jobs. Today, 3 different type of tripleo-ci jobs are >> now >> run across all our clouds, for example there is a centos-7-2-node jobs. >> However, >> tripleo-test-cloud-rh1 is only today setup to accept only tripleo-ci jobs. >> This >> job does not run on tripleo-test-cloud-rh1. >> >> jeblair posted the following statement: >> >> It feels like the tripleo cloud has been grandfathered in its current state >> for a while. I'd just like to make sure we're being fair to everyone. So >> if >> tripleo wants to run tripleo jobs, then i think we should move it to 3rd >> party >> ci. I think that's a fine choice and we can continue to work together >> (please!) but with better division of reponsibilities. Or, if we want to >> revise the idea of a multi-provider hardware platform that's available for >> all >> openstack projects, i'm game for that. It would be great, but more work. >> >> Should we continue the push to move tripleo-test-cloud-rh1 to 3rd party CI >> (removing from nodepool.o.o) or do we start enabling more jobs on >> tripleo-test-cloud-rh1 bringing the cloud even more into openstack-infra? >> >> My personal thoughts, as somebody who's been working on it for the last 4 >> months, I still feel tripleo-test-cloud-rh1 should move to 3rd party CI. >> However, with the work done in the last 4 months, I believe >> tripleo-test-cloud-rh1 _could_ start running additional jobs based on the >> work >> above. >> >> [1] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088988.html >> [2] >> http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-08-05.log.html#t2016-08-05T23:07:35 > > I'd like to provide some additional context about where we'd like to > go with the CI jobs running on all cloud providers. We've added > centos7 2-node multinode jobs, and we have some single node jobs as > well. What I'd like clarification around is there is > hesitation/concern around TripleO building out additional mutlinode > jobs that run on any cloud. > > I'd like to continue down this path of adding more CI jobs that can > run on any nodepool managed cloud provider. To do so, we'd need to > have 3+ node jobs. A 3 node job would let us test with an undercloud > and a 2 node overcloud, while a 4 node job would let us test with an > undercloud and a 3 node overcloud so we could test HA. 4 node HA jobs > seem to work fine in my testing that I've done on > tripleo-test-cloud-rh2, using the same network setup and infra images > that we do for the 2 node job. > > tripleo-ci can't move entirely to using these types of multinode jobs > though, because they do not exercise the nova/ironic deployment during > the tests. So, there will always be some subset of tripleo-ci jobs > that still need tripleo-test-cloud-rh2{1,2} as those clouds are > configured with the necessary OpenStack features enabled to allow for > pxe booting of instances. > > I'm certainly open and supportive of the idea of running all job types > on the tripleo clouds. The work Paul has done should make this > possible. rh2 is on "loan" to the TripleO project team while rh1 moved > datacenters. But, I'm told it can become permanent if there is a need. > Keep in mind this is not a single cloud with 2 regions. rh1 and rh2 > are 2 separate single region clouds, however they offer the equivalent > functionality to run all tripleo-ci and infra job types (barring just > a few outstanding bugs/configuration issues). So, there is failover > for tripleo-ci jobs (if we re-enable rh2). > > At the same time, if there are concerns about capacity from Infra's > perspective as we'd like to build out to at least 4-node multinode > jobs, then I think it makes good sense to open up rh1/rh2 to all job > types, so that they help with the capacity. > In that case, it would not make sense to move tripleo-ci to be a 3rd > party CI setup. > > If there is plenty of capacity on the existing cloud providers and no > concerns about building out additional multinode jobs, then there > might not be a huge benefit in enabling all job types on rh1/rh2. If > that's the case then we might as well just move them to 3rd party CI. > > I suppose it's also possible that we might be pushing too strongly > down the multinode path? Is the general concensus in infra that they'd > like to help enable project teams to eventually add 3 and 4 (and maybe > more) node multinode jobs? If not, then I agree it probably does make > sense to move most of tripleo-ci to be 3rd party, and we'd just keep > our existing check jobs as-is instead of proposing new ones. > > My point is: if there's concensus around TripleO building out to 3+ > node multinode jobs, then I think tripleo-ci should stay where it is > so that we can also work towards contributing to available capacity, > etc.
I think there are some values to test 3+ node multinode jobs: - Testing High-Availability: we currently have pacemaker scenarios, that require at least 3 controllers and 1 compute to test something realistic. Though we have nothing in our tests that disable a controller and run tests again. So unless we have it, I think we could run our tests on a single controller node. - Testing Nova live migration (even if it's tested by devstack) in TripleO, to validate configuration that makes it possible. - Testing 3+ means at least 2 nodes for overcloud, which is the first realistic way to test a multi-node overcloud, and validate TripleO Orchestration, for both initial deployment and upgrades. We are an installer, we rely on orchestration and configuration management, so I think we really need this minimal architecture. Regarding the contribution to OpenStack Infra to give some capacity, it would be certainly fair from us to do it, big +1. > If there is not, and tripleo-ci just keeps adding more jobs that will > only run on our own clouds, then I agree with Paul and Ben that > tripleo-ci feels more naturally to be 3rd party CI. > > -- > -- James Slagle > -- > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
