Hello, Status update for Dec, 5-6 - we're on track.
CentOS7 based ISO looks good, we are building it on product CI using the same jobs as for CentOS6. ISO #258 passed nightly swarm with a slightly reduced coverage (69%). We've got one known issue [0] that affects some jobs when run on a loaded slave (there is a fix [1]), and issues on QA side. Smoke and BVT tests also green [2], however, we've got interference from Ubuntu upstream here, and had to disable trusty-proposed channel to get tests passed. More details in a bug [3]. According to our plan [4] we've got over decision point #4 with a decision to go with CentOS7. Please note that merge freeze still taken place, it will be explicitly notified when it's lifted. [0] https://bugs.launchpad.net/mos/+bug/1523117 [1] https://review.openstack.org/#/c/253843/ [2] https://product-ci.infra.mirantis.net/job/8.0.test_all/245/ [3] https://bugs.launchpad.net/fuel/+bug/1523092 [4] http://lists.openstack.org/pipermail/openstack-dev/2015-December/081026.html On Tue, 1 Dec 2015 13:48:00 -0800 Dmitry Borodaenko <dborodae...@mirantis.com> wrote: > With bit more details, I hope this covers all the risks and decision > points now. > > First of all, current list of outstanding commits: > https://etherpad.openstack.org/p/fuel_on_centos7 > > The above list has two sections: backwards compatible changes that can > be merged one at a time even if the rest of CentOS7 support isn't > merged, and backwards incompatible changes that break support for > CentOS6 and must be merged (and, if needed, reverted) all at once. > > Decision point 1: FFE for CentOS7 > > CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can > it be allowed a Feature Freeze Exception? So far, the disruption of > the Fuel development process implied by the proposed merge plan is > acceptable, if anything goes wrong and we become unable to have a > stable ISO with merged CentOS7 support on Monday, December 7, the FFE > will be revoked. > > Wed, Dec 2: Merge party > > Merge party before 8.0 FF, we should do our best to merge all > remaining feature commits before end of day (including backwards > compatible CentOS7 support commits), without breaking the build too > much. > > At the end of the day we'll start a swarm test over the result of the > merge party, and we expect QA to analyze and summarize the results by > 17:00 MSK (6:00 PST) on Thu Dec 3. > > Risk 1: Merge party breaks the build > > If there is a large regression in swarm pass percentage, we won't be > able to afford a merge freeze which is necessary to merge CentOS7 > support, we'll have to be merging bugfixes until swarm test pass rate > is back around 70%. > > Risk 2: More features get FFE > > If some essential 8.0 features are not completely merged by end of day > Wed Dec 2 and are granted FFE, merging the remaining commits can > interfere with merging CentOS7 support, not just from merge conflicts > perspective, but also invalidating swarm results and making it > practically impossible to bisect and attribute potential regressions. > > Thu, Dec 3: Start merge freeze for CentOS7 > > Decision point 2: Other FFEs > > In the morning MSK time, we will assess Risk 2 and decide what to do > with the other FFEs. The options are: integrate remaining commits into > CentOS7 merge plan, block remaining commits until Monday, revoke > CentOS7 FFE. > > If the decision is to go ahead with CentOS7 merge, we announce merge > freeze for all git repositories that go into Fuel ISO, and spend the > rest of the day rebasing and cleaning up the rest of the CentOS7 > commits to make sure they're all in mergeable state by the end of the > day. The outcome of this work must be a custom ISO image with all > remaining commits, with additional requirement that it must not use > Jenkins job parameters (only patches to fuel-main that change default > repository paths) to specify all required package repositories. This > will validate the proposed fuel-main patches and ensure that no > unmerged package changes are used to produce the ISO. > > Decision point 3: Swarm pass rate > > After swarm results from Wed are available, we will assess the Risk 1. > If the pass rate regression is significant, CentOS7 FFE is revoked and > merge freeze is lifted. If regression is acceptable, we proceed with > merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4. > > Fri, Dec 4: Merge and test CentOS7 > > The team will have until 17:00 MSK to produce a non-custom ISO that > passes BVT and can be run through swarm. > > Sat, Dec 5: Assess CentOS7 swarm and bugfix > > First of all, someone from CI and QA teams should commit to monitoring > the CentOS7 swarm run and report the results as soon as possible. > Based on the results (which once again must be available by 17:00 > MSK), we can decide on the final step of the plan. > > Decision point 4: Keep or revert > > If CentOS7 based swarm shows significant regression, we have to spend > the rest of the weekend including Sunday reverting all CentOS7 commits > that were merged during merge freeze. Once revert is completed, we > will lift the merge freeze. > > If the regression is acceptable, we lift the merge freeze straight > away and proceed with bugfixing as usual. At this point CI team will > need to update the Fuel ISO used for deployment tests in our CI to > this same ISO. > > One way or the other, we will be able to resume bugfixing on Monday > morning MSK time, and will have lost 2 business days (Thu-Fri) during > which we won't be able to merge bugfixes. In addition to that, someone > from QA and everyone from CentOS7 support team has to work on > Saturday, and someone from CI will have to work a few hours on Sunday. > -- Thanks, Dmitry Teselkin __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
