As we agreed, we have switched ISO builds to latest CentOS 7.2 snapshots. You can see now that each ISO build (see for ex. [1]) produces several *_id.txt artifacts. Note that centos_mirror_id.txt points to CentOS snapshot at http://mirror.fuel-infra.org/pkgs/
BVT test is stable, see [2], and nightly system tests results are basically the same as they were before. [1] https://ci.fuel-infra.org/job/9.0-community.all/3868/ [2] https://ci.fuel-infra.org/view/ISO/job/9.0.fuel_community.ubuntu.bvt_2/ On Thu, Mar 3, 2016 at 3:46 AM, Dmitry Borodaenko <dborodae...@mirantis.com> wrote: > Thanks for the detailed explanation, very helpful! > > Considering that this change is atomic and easily revertable, lets > proceed with the change, the sooner we do that the more time we'll have > to confirm that there is no impact and revert if necessary. > > -- > Dmitry Borodaenko > > On Thu, Mar 03, 2016 at 03:40:22AM +0300, Aleksandra Fedorova wrote: >> Hi, >> >> let me add some details about the change: >> >> 1) There are two repositories used to build Fuel ISO: base OS >> repository [1], and mos repository [2], where we put Fuel dependencies >> and packages we rebuild due to certain version requirements. >> >> The CentOS 7.2 feature is related to the upstream repo only. Packages >> like RabbitMQ, MCollective, Puppet, MySQL and PostgreSQL live in mos >> repository, which has higher priority then upstream. >> >> I think we need to setup a separate discussion about our policy >> regarding these packages, but for now they are fixed and won't be >> updated by CentOS 7.2 switch. >> >> 2) This change doesn't affect Fuel codebase. >> >> The upstream mirror we use for ISO build is controlled by environment >> variable which is set via Jenkins [3] and can be changed anytime. >> >> As we have daily snapshots of CentOS repository available at [4], in >> case of regression in upstream we can pin our builds to stable >> snapshot and work on the issue without blocking the main development >> flow. > > Please make sure that the current snapshot of CentOS 7.1 is not rotated > away so that we don't loose the point we can revert to. > >> 3) The "improve snapshotting" work item which is at the moment in >> progress, will prevent any possibility to "accidentally" migrate to >> CentOS 7.3, when it becomes available. >> Thus the only changes which we can fetch from upstream are changes >> which are published to updates/ component of CentOS 7.2 repo. >> >> >> As latest BVT on master is green >> https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/69/ >> I think we should proceed with Jenkins reconfiguration [5] and switch >> to latest snapshots by default. >> >> [1] currently http://vault.centos.org/7.1.1503/ >> [2] >> http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7-fuel/os/x86_64/ >> [3] >> https://github.com/fuel-infra/jenkins-jobs/blob/76b5cdf1828b7db1957f7967180d20be099b0c63/common/scripts/all.sh#L84 >> [4] http://mirror.fuel-infra.org/pkgs/ >> [5] https://review.fuel-infra.org/#/c/17712/ >> >> On Wed, Mar 2, 2016 at 9:22 PM, Mike Scherbakov >> <mscherba...@mirantis.com> wrote: >> > It is not just about BVT. I'd suggest to monitor situation overall, >> > including failures of system tests [1]. If we see regressions there, or >> > some >> > test cases will start flapping (what is even worse), then we'd have to >> > revert back to CentOS 7.1. >> > >> > [1] https://github.com/openstack/fuel-qa >> > >> > On Wed, Mar 2, 2016 at 10:16 AM Dmitry Borodaenko >> > <dborodae...@mirantis.com> >> > wrote: >> >> >> >> I agree with Mike's concerns, and propose to make these limitations (4 >> >> weeks before FF for OS upgrades, 2 weeks for upgrades of key >> >> dependencies -- RabbitMQ, MCollective, Puppet, MySQL, PostgreSQL, >> >> anything else?) official for 10.0/Newton. >> >> >> >> For 9.0/Mitaka, it is too late to impose them, so we just have to be >> >> very careful and conservative with this upgrade. First of all, we need >> >> to have a green BVT before and after switching to the CentOS 7.2 repo >> >> snapshot, so while I approved the spec, we can't move forward with this >> >> until BVT is green again, and right now it's red: >> >> >> >> https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/ >> >> >> >> If we get it back to green but it becomes red after the upgrade, you >> >> must switch back to CentOS 7.1 *immediately*. If you are able to stick >> >> to this plan, there is still time to complete the transition today >> >> without requiring an FFE. >> >> >> >> -- >> >> Dmitry Borodaenko >> >> >> >> >> >> On Wed, Mar 02, 2016 at 05:53:53PM +0000, Mike Scherbakov wrote: >> >> > Formally, we can merge it today. Historically, every update of OS caused >> >> > us >> >> > instability for some time: from days to a couple of month. >> >> > Taking this into account and number of other exceptions requested, >> >> > overall >> >> > stability of code, my opinion would be to postpone this to 10.0. >> >> > >> >> > Also, I'd suggest to change the process, and have freeze date for all OS >> >> > updates no later than a month before official FF date. This will give us >> >> > time to stabilize, and ensure that base on which all new code is being >> >> > developed is stable when approaching FF. >> >> > >> >> > I'd also propose to have freeze for major upgrades of 3rd party packages >> >> > no >> >> > later than 2 weeks before FF, which Fuel depends heavily upon. For >> >> > instance, such will include RabbitMQ, MCollective, Puppet. >> >> > >> >> > On Wed, Mar 2, 2016 at 7:34 AM Igor Marnat <imar...@mirantis.com> wrote: >> >> > >> >> > > Igor, >> >> > > couple of points from my side. >> >> > > >> >> > > CentOS 7.2 will be getting updates for several more months, and we >> >> > > have >> >> > > snapshots and all the mechanics in place to switch to the next version >> >> > > when >> >> > > needed. >> >> > > >> >> > > Speaking of getting this update into 9.0, we actually don't need FFE, >> >> > > we >> >> > > can merge remaining staff today. It has enough reviews, so if you add >> >> > > your >> >> > > +1 today, we don't need FFE. >> >> > > >> >> > > https://review.openstack.org/#/c/280338/ >> >> > > https://review.fuel-infra.org/#/c/17400/ >> >> > > >> >> > > >> >> > > >> >> > > Regards, >> >> > > Igor Marnat >> >> > > >> >> > > On Wed, Mar 2, 2016 at 6:23 PM, Dmitry Teselkin >> >> > > <dtesel...@mirantis.com> >> >> > > wrote: >> >> > > >> >> > >> Igor, >> >> > >> >> >> > >> Your statement about updates for 7.2 isn't correct - it will receive >> >> > >> updates, because it's the latest release ATM. There is *no* pinning >> >> > >> inside >> >> > >> ISO, and the only place where it was 8.0 were docker containers just >> >> > >> because we had to workaround some issues. But there are no docker >> >> > >> containers in 9.0, so that's not the case. >> >> > >> The proposed solution to switch to CentOS-7.2 in fact is based on >> >> > >> selecting the right snapshot with packages. There is no pinning in >> >> > >> ISO (it >> >> > >> was in earlier versions of the spec but was removed). >> >> > >> >> >> > >> On Wed, Mar 2, 2016 at 6:11 PM, Igor Kalnitsky >> >> > >> <ikalnit...@mirantis.com> >> >> > >> wrote: >> >> > >> >> >> > >>> Dmitry, Igor, >> >> > >>> >> >> > >>> > Very important thing is that CentOS 7.1 which master node is based >> >> > >>> > now >> >> > >>> > don't get updates any longer. >> >> > >>> >> >> > >>> If you are using "fixed" release you must be ready that you won't >> >> > >>> get >> >> > >>> any updates. So with CentOS 7.2 the problem still the same. >> >> > >>> >> >> > >>> However, let's wait for Fuel PTL decision. I only shared my POV: >> >> > >>> that's not a critical feature, and taking into account the risks of >> >> > >>> regression - I'd prefer to do not accept it in 9.0. >> >> > >>> >> >> > >>> Regards, >> >> > >>> Igor >> >> > >>> >> >> > >>> On Wed, Mar 2, 2016 at 4:42 PM, Igor Marnat <imar...@mirantis.com> >> >> > >>> wrote: >> >> > >>> > Igor, >> >> > >>> > please note that this is pretty much not like update of master >> >> > >>> > node >> >> > >>> which we >> >> > >>> > had in 8.0. This is minor _update_ of CentOS from 7.1 to 7.2 which >> >> > >>> > team >> >> > >>> > tested for more than 2 months already. >> >> > >>> > >> >> > >>> > We don't expect it to require any additional efforts from core or >> >> > >>> > qa >> >> > >>> team. >> >> > >>> > >> >> > >>> > Very important thing is that CentOS 7.1 which master node is based >> >> > >>> > now >> >> > >>> don't >> >> > >>> > get updates any longer. Updates are only provided for CentOS 7.2. >> >> > >>> > >> >> > >>> > So we'll have to switch CentOS 7.1 to CentOS 7.2 anyways. >> >> > >>> > >> >> > >>> > We can do it now for more or less free, later in release cycle for >> >> > >>> higher >> >> > >>> > risk and QA efforts and after the release for 2x price because of >> >> > >>> additional >> >> > >>> > QA cycle we'll need to pass through. >> >> > >>> > >> >> > >>> > >> >> > >>> > >> >> > >>> > Regards, >> >> > >>> > Igor Marnat >> >> > >>> > >> >> > >>> > On Wed, Mar 2, 2016 at 2:57 PM, Dmitry Teselkin < >> >> > >>> dtesel...@mirantis.com> >> >> > >>> > wrote: >> >> > >>> >> >> >> > >>> >> Hi Igor, >> >> > >>> >> >> >> > >>> >> Postponing this till Fuel 10 means we have to elaborate a plan to >> >> > >>> >> do >> >> > >>> such >> >> > >>> >> upgrade for Fuel 9 after the release - the underlying system will >> >> > >>> >> not >> >> > >>> get >> >> > >>> >> updated on it's own, and the security issues will not close >> >> > >>> themselves. The >> >> > >>> >> problem here is that such upgrade of deployed master node >> >> > >>> >> requires a >> >> > >>> lot of >> >> > >>> >> QA work also. >> >> > >>> >> >> >> > >>> >> Since we are not going to update package we build on our own >> >> > >>> >> (they >> >> > >>> still >> >> > >>> >> targeted 7.1) switching master node base to CentOS-72 is not that >> >> > >>> dangerous >> >> > >>> >> as it seems. >> >> > >>> >> >> >> > >>> >> On Wed, Mar 2, 2016 at 1:54 PM, Igor Kalnitsky < >> >> > >>> ikalnit...@mirantis.com> >> >> > >>> >> wrote: >> >> > >>> >>> >> >> > >>> >>> Hey Dmitry, >> >> > >>> >>> >> >> > >>> >>> No offence, but I rather against that exception. We have too >> >> > >>> >>> many >> >> > >>> >>> things to do in Mitaka, and moving to CentOS 7.2 means >> >> > >>> >>> >> >> > >>> >>> * extra effort from core team >> >> > >>> >>> * extra effort from qa team >> >> > >>> >>> >> >> > >>> >>> Moreover, it might block development by introducing >> >> > >>> >>> unpredictable >> >> > >>> >>> regressions. Remember 8.0? So I think it'd be better to reduce >> >> > >>> >>> risk >> >> > >>> of >> >> > >>> >>> regressions that affects so many developers by postponing CentOS >> >> > >>> >>> 7.2 >> >> > >>> >>> till Fuel 10. >> >> > >>> >>> >> >> > >>> >>> Thanks, >> >> > >>> >>> Igor >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> On Mon, Feb 29, 2016 at 7:13 PM, Dmitry Teselkin < >> >> > >>> dtesel...@mirantis.com> >> >> > >>> >>> wrote: >> >> > >>> >>> > I'd like to ask for a feature freeze exception for switching >> >> > >>> >>> > to >> >> > >>> >>> > CentOS-7.2 >> >> > >>> >>> > feature [0]. >> >> > >>> >>> > >> >> > >>> >>> > CentOS-7.2 ISO's have been tested periodically since the >> >> > >>> >>> > beginning >> >> > >>> of >> >> > >>> >>> > the >> >> > >>> >>> > year, and all major issues were addressed / fixed at the >> >> > >>> >>> > moment. >> >> > >>> During >> >> > >>> >>> > the >> >> > >>> >>> > last weekend I've made 70 BVT runs to verify that the >> >> > >>> >>> > solution >> >> > >>> [2] for >> >> > >>> >>> > the >> >> > >>> >>> > last issue - e1000 transmit unit hangs works. And it works, 0 >> >> > >>> tests of >> >> > >>> >>> > 35 >> >> > >>> >>> > failed [3]. >> >> > >>> >>> > >> >> > >>> >>> > Benefits of switching to CentOS-7.2 are quite obvious - we >> >> > >>> >>> > will >> >> > >>> return >> >> > >>> >>> > to >> >> > >>> >>> > latest supported CentOS release, will fix a lot of bugs / >> >> > >>> >>> > security >> >> > >>> >>> > issues >> >> > >>> >>> > [4] and will make further updates easier. >> >> > >>> >>> > >> >> > >>> >>> > Risk of regression still exists, but it's quite low, 35 >> >> > >>> >>> > successful >> >> > >>> BVTs >> >> > >>> >>> > can't be wrong. >> >> > >>> >>> > >> >> > >>> >>> > To finish that feature the following should be done: >> >> > >>> >>> > * review and merge e1000 workaround [2] >> >> > >>> >>> > * review and merge spec [0] >> >> > >>> >>> > * review and merge request that switches build CI to >> >> > >>> >>> > CentOS-7.2 [5] >> >> > >>> >>> > >> >> > >>> >>> > I expect the last day it could be done is March, 4. >> >> > >>> >>> > >> >> > >>> >>> > [0] https://review.openstack.org/#/c/280338/ >> >> > >>> >>> > [1] https://bugs.launchpad.net/fuel/+bug/1526544 >> >> > >>> >>> > [2] https://review.openstack.org/#/c/285306/ >> >> > >>> >>> > [3] >> >> > >>> https://etherpad.openstack.org/p/r.1c4cfee8185326d6922d6c9321404350 >> >> > >>> >>> > [4] >> >> > >>> https://etherpad.openstack.org/p/r.a7fe0b575d891ed81206765fa5be6630 >> >> > >>> >>> > [5] https://review.fuel-infra.org/#/c/17400/ >> >> > >>> >>> > >> >> > >>> >>> > >> >> > >>> >>> > -- >> >> > >>> >>> > Thanks, >> >> > >>> >>> > Dmitry Teselkin >> >> > >>> >>> > Mirantis >> >> > >>> >>> > http://www.mirantis.com >> >> > >>> >>> > >> >> > >>> >>> > >> >> > >>> >>> > >> >> > >>> >> >> > >>> __________________________________________________________________________ >> >> > >>> >>> > OpenStack Development Mailing List (not for usage questions) >> >> > >>> >>> > Unsubscribe: >> >> > >>> >>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > >>> >>> > >> >> > >>> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >>> >>> > >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >> >> > >>> __________________________________________________________________________ >> >> > >>> >>> OpenStack Development Mailing List (not for usage questions) >> >> > >>> >>> Unsubscribe: >> >> > >>> >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > >>> >>> >> >> > >>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> -- >> >> > >>> >> Thanks, >> >> > >>> >> Dmitry Teselkin >> >> > >>> >> Mirantis >> >> > >>> >> http://www.mirantis.com >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> > >>> __________________________________________________________________________ >> >> > >>> >> OpenStack Development Mailing List (not for usage questions) >> >> > >>> >> Unsubscribe: >> >> > >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >>> >> >> >> > >>> > >> >> > >>> > >> >> > >>> > >> >> > >>> >> >> > >>> __________________________________________________________________________ >> >> > >>> > OpenStack Development Mailing List (not for usage questions) >> >> > >>> > Unsubscribe: >> >> > >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >>> > >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> __________________________________________________________________________ >> >> > >>> OpenStack Development Mailing List (not for usage questions) >> >> > >>> Unsubscribe: >> >> > >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >>> >> >> > >> >> >> > >> >> >> > >> >> >> > >> -- >> >> > >> Thanks, >> >> > >> Dmitry Teselkin >> >> > >> Mirantis >> >> > >> http://www.mirantis.com >> >> > >> >> >> > >> >> >> > >> __________________________________________________________________________ >> >> > >> OpenStack Development Mailing List (not for usage questions) >> >> > >> Unsubscribe: >> >> > >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >> >> >> > >> >> >> > > >> >> > > __________________________________________________________________________ >> >> > > OpenStack Development Mailing List (not for usage questions) >> >> > > Unsubscribe: >> >> > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > >> >> > -- >> >> > Mike Scherbakov >> >> > #mihgen >> >> >> >> > >> >> > __________________________________________________________________________ >> >> > OpenStack Development Mailing List (not for usage questions) >> >> > Unsubscribe: >> >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> >> __________________________________________________________________________ >> >> OpenStack Development Mailing List (not for usage questions) >> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > -- >> > Mike Scherbakov >> > #mihgen >> > >> > __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> >> >> -- >> Aleksandra Fedorova >> CI Team Lead >> bookwar >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Aleksandra Fedorova CI Team Lead bookwar __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev