Re: [openstack-dev] [OpenStack-Infra] [TripleO] promoting devtest_seed and devtest_undercloud to voting, + experimental queue for nova/neutron etc.
On 15 February 2014 09:58, Sean Dague s...@dague.net wrote: Lastly, I'm going to propose a merge to infra/config to put our undercloud story (which exercises the seed's ability to deploy via heat with bare metal) as a check experimental job on our dependencies (keystone, glance, nova, neutron) - if thats ok with those projects? -Rob My biggest concern with adding this to check experimental, is the experimental results aren't published back until all the experimental jobs are done. If we add a new pipeline - https://review.openstack.org/#/c/73863/ - then we can avoid that. We've seen really substantial delays, plus a 5 day complete outage a week ago, on the tripleo cloud. I'd like to see that much more proven before it starts to impact core projects, even in experimental. I believe that with a new pipeline it won't impact core projects at all. The outage, FWIW, was because I deleted the entire cloud, at the same time that we had a firedrill with some other upstream-of-us issue (I forget the exact one). The multi-region setup we're aiming for should mitigate that substantially :) -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Infra] [TripleO] promoting devtest_seed and devtest_undercloud to voting, + experimental queue for nova/neutron etc.
On 15 February 2014 12:21, James E. Blair jebl...@openstack.org wrote: You won't end up with -1's everywhere, you'll end up with jobs stuck in the queue indefinitely, as we saw when the tripleo cloud failed recently. What's worse is that now that positive check results are required for enqueuing into the gate, you will also not be able to merge anything. Ok. So the cost of voting [just in tripleo] would be that a) [tripleo] infrastructure failures and b) breakage from other projects - both things that can cause checks to fail, would stall all tripleo landings until rectified, or until voting is turned off via a change to config which makes this infra's problem. Hmm - so from a tripleo perspective, I think we're ok with this - having a clear indication that 'this is ok' is probably more important to us right now than the more opaque thing we have now where we have to expand every jenkins comment to be sure. But- will infra be ok, if we end up having a firedrill 'please make this nonvoting' change to propose? -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Infra] [TripleO] promoting devtest_seed and devtest_undercloud to voting, + experimental queue for nova/neutron etc.
Sean Dague s...@dague.net writes: On 02/14/2014 03:43 PM, Robert Collins wrote: Thanks to a massive push this week, both the seed *and* undercloud jobs are now passing on tripleo-gate nodes, but they are not yet voting. I'd kind of like to get them voting on tripleo jobs (check only). We don't have 2 clouds yet, so if the tripleo ci-cloud suffers a failure, we'd have -1's everywhere. I think this would be an ok tradeoff (its check after all), but I'd like -infra admin folks opinion on this - would it cause operational headaches for you, over and above the current risks w/ the tripleo-ci cloud? You won't end up with -1's everywhere, you'll end up with jobs stuck in the queue indefinitely, as we saw when the tripleo cloud failed recently. What's worse is that now that positive check results are required for enqueuing into the gate, you will also not be able to merge anything. OTOH - we actually got passing ops with a fully deployed virtual cloud - which is awesome. Great! :) Now we need to push through to having the overcloud deploy tests pass, then the other scenarios we depend on - upgrades w/rebuild, and we'll be in good shape to start optimising (pre-heated clouds, local distro mirrors etc) and broadening (other distros ...). Lastly, I'm going to propose a merge to infra/config to put our undercloud story (which exercises the seed's ability to deploy via heat with bare metal) as a check experimental job on our dependencies (keystone, glance, nova, neutron) - if thats ok with those projects? -Rob My biggest concern with adding this to check experimental, is the experimental results aren't published back until all the experimental jobs are done. We've seen really substantial delays, plus a 5 day complete outage a week ago, on the tripleo cloud. I'd like to see that much more proven before it starts to impact core projects, even in experimental. Until the tripleo cloud is multi-region, HA, and has a proven track record of reliability, we can't have jobs that run on its nodes in any pipeline for any non-tripleo project, for those reasons. I do look forward to when that is the case. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
