Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Mon, Aug 21, 2017 at 10:43:07AM +1200, Steve Baker wrote: > On Thu, Aug 17, 2017 at 4:13 PM, Steve Baker wrote: > > > > > > > On Thu, Aug 17, 2017 at 10:47 AM, Emilien Macchi > > wrote: > > > >> > >> > Problem #3: from Ocata to Pike: all container

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Thu, Aug 17, 2017 at 4:13 PM, Steve Baker wrote: > > > On Thu, Aug 17, 2017 at 10:47 AM, Emilien Macchi > wrote: > >> >> > Problem #3: from Ocata to Pike: all container images are >> > uploaded/specified, even for services not deployed >> >

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On 17.8.2017 00:47, Emilien Macchi wrote: Here's an update on the situation. On Tue, Aug 15, 2017 at 6:33 PM, Emilien Macchi wrote: Problem #1: Upgrade jobs timeout from Newton to Ocata https://bugs.launchpad.net/tripleo/+bug/1702955 [...] - revert distgit patch in RDO:

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Thu, Aug 17, 2017 at 10:47 AM, Emilien Macchi wrote: > > > Problem #3: from Ocata to Pike: all container images are > > uploaded/specified, even for services not deployed > > https://bugs.launchpad.net/tripleo/+bug/1710992 > > The CI jobs are timeouting during the upgrade

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Wed, Aug 16, 2017 at 3:47 PM, Emilien Macchi wrote: > Here's an update on the situation. > > On Tue, Aug 15, 2017 at 6:33 PM, Emilien Macchi wrote: >> Problem #1: Upgrade jobs timeout from Newton to Ocata >>

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

Here's an update on the situation. On Tue, Aug 15, 2017 at 6:33 PM, Emilien Macchi wrote: > Problem #1: Upgrade jobs timeout from Newton to Ocata > https://bugs.launchpad.net/tripleo/+bug/1702955 [...] - revert distgit patch in RDO: https://review.rdoproject.org/r/8575 -

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Tue, Aug 15, 2017 at 11:06:20PM -0400, Wesley Hayutin wrote: > On Tue, Aug 15, 2017 at 9:33 PM, Emilien Macchi wrote: > > > So far, we're having 3 critical issues, that we all need to address as > > soon as we can. > > > > Problem #1: Upgrade jobs timeout from Newton to

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Wed, Aug 16, 2017 at 3:17 AM, Bogdan Dobrelya wrote: > We could limit the upstream multinode jobs scope to only do upgrade > testing of a couple of the services deployed, like keystone and nova and > neutron, or so. That would be a huge regression in our CI. Strong -2 on

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Wed, Aug 16, 2017 at 12:37 AM, Marios Andreou wrote: > For Newton to Ocata, is it consistent which clouds we are timing out on? It's not consistent but the rate is very high: http://cistatus.tripleo.org/ gate-tripleo-ci-centos-7-multinode-upgrades - 30 % of success this

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On 16.08.2017 3:33, Emilien Macchi wrote: > So far, we're having 3 critical issues, that we all need to address as > soon as we can. > > Problem #1: Upgrade jobs timeout from Newton to Ocata > https://bugs.launchpad.net/tripleo/+bug/1702955 > Today I spent an hour to look at it and here's what

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On 16.08.2017 5:06, Wesley Hayutin wrote: > > > On Tue, Aug 15, 2017 at 9:33 PM, Emilien Macchi > wrote: > > So far, we're having 3 critical issues, that we all need to address as > soon as we can. > > Problem #1: Upgrade jobs

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Wed, Aug 16, 2017 at 4:33 AM, Emilien Macchi wrote: > So far, we're having 3 critical issues, that we all need to address as > soon as we can. > > Problem #1: Upgrade jobs timeout from Newton to Ocata > https://bugs.launchpad.net/tripleo/+bug/1702955 > Today I spent an

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

On Tue, Aug 15, 2017 at 9:33 PM, Emilien Macchi wrote: > So far, we're having 3 critical issues, that we all need to address as > soon as we can. > > Problem #1: Upgrade jobs timeout from Newton to Ocata > https://bugs.launchpad.net/tripleo/+bug/1702955 > Today I spent an

[openstack-dev] [tripleo] critical situation with CI / upgrade jobs

So far, we're having 3 critical issues, that we all need to address as soon as we can. Problem #1: Upgrade jobs timeout from Newton to Ocata https://bugs.launchpad.net/tripleo/+bug/1702955 Today I spent an hour to look at it and here's what I've found so far: depending on which public cloud we're