Re: win64-rev2 cutover - Aurora and Beta timing
mozilla-aurora and comm-aurora are now building against the win64-rev2 build image. mozilla-beta and comm-beta will follow suit next week (for beta 1), as planned. Thanks, John On 13-12-03 2:49 PM, John Hopkins wrote: > Status of the Windows build image upgrade from [1]: > > * mozilla-central and comm-central branches are building only on win64-rev2 > * mozilla-inbound and b2g-inbound remain on a mixed rev1/rev2 pool to > ensure an adequate number of build slaves to meet its high checkin load > * 24 win64-rev1 build machines are left to reimage, pending cutover of > aurora, beta, release, esr branches. > * Firefox desktop nightlies are building on win64-rev2 since Tuesday, > November 27 > > > Given that builds on the new Windows image have been running smoothly > for Nightlies, projects, inbound, etc., I would like to cut over Aurora > builds on Thursday and Beta builds next week for the Beta 1 builds. > > Rail Aliiev has tested a beta build (of the current aurora branch) in > staging using the new build image and did not run into any issues. > > Please let me know if you have any questions or concerns with these > cutover plans before Thursday. > > Thanks, > John > > [1] https://bugzilla.mozilla.org/show_bug.cgi?id=781277#c58 > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
win64-rev2 cutover - Aurora and Beta timing
Status of the Windows build image upgrade from [1]: * mozilla-central and comm-central branches are building only on win64-rev2 * mozilla-inbound and b2g-inbound remain on a mixed rev1/rev2 pool to ensure an adequate number of build slaves to meet its high checkin load * 24 win64-rev1 build machines are left to reimage, pending cutover of aurora, beta, release, esr branches. * Firefox desktop nightlies are building on win64-rev2 since Tuesday, November 27 Given that builds on the new Windows image have been running smoothly for Nightlies, projects, inbound, etc., I would like to cut over Aurora builds on Thursday and Beta builds next week for the Beta 1 builds. Rail Aliiev has tested a beta build (of the current aurora branch) in staging using the new build image and did not run into any issues. Please let me know if you have any questions or concerns with these cutover plans before Thursday. Thanks, John [1] https://bugzilla.mozilla.org/show_bug.cgi?id=781277#c58 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
win64-rev2 cutover - ok to migrate mozilla-central to a mixed pool?
Status so far: * The try, mozilla-inbound, b2g-inbound, and try-comm-central branches have been running on a mixed win64-rev1 + win64-rev2 pool since October 31st. * Project branches have been building solely on win64-rev2 since October 10th. * Builds on the cedar and fx-team branches passed smoke testing by QA * Last week, 40 production and 40 try w64-ix-slaves were converted to rev2. That means almost all Try builds are being done on rev2. I just looked at the bug tree for win64-rev2 tracking bug 781277 and pinged RyanVM|sheriffduty and neither indicate any win64-rev2 specific build issues (pending verification of bug 933271). Questions: * Are there any objections to cutting over mozilla-central and comm-central to a mixed win64-rev1 + win64-rev2 pool on Wednesday of this week? * Are we happy with the coverage we have so far, or do we want to do any additional testing before building Nightlies? Thanks, John ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Wanted: Feedback on Windows 64-bit "rev2" cutover options
On 13-10-17 3:22 PM, Armen Zambrano Gasparnian wrote: > Option #3 sounds logical. Based on feedback I received from joduinn, one small adjustment to option #3: phase-in to "try" first and wait for confirmation before phasing-in inbound/central to further reduce risk exposure to those branches. > > What way would sheriffs/devs have to determine which machines are rev2 > and which ones are rev1? > Should they use the info in production_config.py? > - rev2 machines: [1] > - rev2 (try): currently empty [2] Yes, using production_config.py would be a sure way to tell the difference. You can also tell them apart by looking at the build root in a build log: rev1 - e:\builds\moz2_slave rev2 - c:\builds\moz2_slave John > > [1] > http://hg.mozilla.org/build/buildbot-configs/file/production/mozilla/production_config.py#l8 > [2] > http://hg.mozilla.org/build/buildbot-configs/file/production/mozilla/production_config.py#l33 > > On 2013-10-17 2:34 PM, John Hopkins wrote: >> Here are three different proposals to cut over to the new Windows 64-bit >> rev2 (win64-rev2) machines (see >> https://groups.google.com/forum/#!topic/mozilla.dev.platform/zACrUe_JwKw >> for context), along with some of the pros and cons of each approach. >> I would prefer option #3 (gradual phase-in) so long as we're okay with >> having a mixed pool of rev1/rev2 win64 build machines on the same >> branches. Timing will depend on which option we go with. >> >> Please let me know as soon as possible your preference or whether you >> have any other comments/concerns. I will assume option #3 if there are >> no objections by end of day Friday. >> >> In addition, I'd like to have a smoke test performed against the Windows >> builds on one of our project branches (which have already been cut over) >> - fx-team or ux seem like good candidates. I will follow up with the >> QA team. >> >> >> Win64-rev2 Cutover Proposals: >> >> 1) Cut over all of inbound/try/central to win64-rev2 over a weekend. >> Pros: >> - all branches built using the same machine image >> - lowest volume of checkins happen on weekends, so less wait time impact >> Cons: >> - "big bang" upgrade. If someone discovers an issue on Monday, a >> lengthy downtime could be required to resolve it. >> - build wait times on the weekend could be lengthy or require a tree closure >> - staffing weekend work is problematic >> >> >> 2) Cut over inbound/try/central branches one at a time, each early on a >> Monday (over several weeks) >> Pros: >> - more people around to find/fix problems >> Cons: >> - longer wait times >> - not all branches using the same build machine image >> - longer cutover time >> >> >> 3) Gradual phase-in of win64-rev2 machines on inbound/try/central in a >> mixed rev1/rev2 pool >> Pros: >> - limited impact of bustages (due to mixing in a small number of rev2 >> build machines to start and gradually increasing) >> - no impact on wait times (could even improve them slightly since we're >> a bit low on rev1 capacity at the moment) >> - no weekend work required, can be done during the week as time permits >> Cons: >> - mix of rev1/rev2 build machines, mitigated by having exclusively rev2 >> allocated to project branches for testing rev2-specific bustage fixes >> >> >> Note: This will not impact release branches (aurora, beta, release, >> esr). We will cut over to those branches only once win64 rev2 has been >> proven on inbound/try/central. Bug 781277 is the overall tracking bug. >> >> Thanks, >> John >> > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Wanted: Feedback on Windows 64-bit "rev2" cutover options
Here are three different proposals to cut over to the new Windows 64-bit rev2 (win64-rev2) machines (see https://groups.google.com/forum/#!topic/mozilla.dev.platform/zACrUe_JwKw for context), along with some of the pros and cons of each approach. I would prefer option #3 (gradual phase-in) so long as we're okay with having a mixed pool of rev1/rev2 win64 build machines on the same branches. Timing will depend on which option we go with. Please let me know as soon as possible your preference or whether you have any other comments/concerns. I will assume option #3 if there are no objections by end of day Friday. In addition, I'd like to have a smoke test performed against the Windows builds on one of our project branches (which have already been cut over) - fx-team or ux seem like good candidates. I will follow up with the QA team. Win64-rev2 Cutover Proposals: 1) Cut over all of inbound/try/central to win64-rev2 over a weekend. Pros: - all branches built using the same machine image - lowest volume of checkins happen on weekends, so less wait time impact Cons: - "big bang" upgrade. If someone discovers an issue on Monday, a lengthy downtime could be required to resolve it. - build wait times on the weekend could be lengthy or require a tree closure - staffing weekend work is problematic 2) Cut over inbound/try/central branches one at a time, each early on a Monday (over several weeks) Pros: - more people around to find/fix problems Cons: - longer wait times - not all branches using the same build machine image - longer cutover time 3) Gradual phase-in of win64-rev2 machines on inbound/try/central in a mixed rev1/rev2 pool Pros: - limited impact of bustages (due to mixing in a small number of rev2 build machines to start and gradually increasing) - no impact on wait times (could even improve them slightly since we're a bit low on rev1 capacity at the moment) - no weekend work required, can be done during the week as time permits Cons: - mix of rev1/rev2 build machines, mitigated by having exclusively rev2 allocated to project branches for testing rev2-specific bustage fixes Note: This will not impact release branches (aurora, beta, release, esr). We will cut over to those branches only once win64 rev2 has been proven on inbound/try/central. Bug 781277 is the overall tracking bug. Thanks, John ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Migrating to Win64 rev2 machines
On 13-10-10 4:25 PM, John Hopkins wrote: > > Project branches are cutting over to the new win64-rev2 build machines. Project branch builds have run overnight on the new build machines and are looking good. We will continue migrating more branches next week. John > On 13-10-09 5:22 PM, John Hopkins wrote: >> Release Engineering is migrating the Windows 64-bit _build_ machines to >> a new, managed machine image which will decrease the amount of >> administrative overhead needed to deploy new instances, make image >> changes, etc. >> >> It uses the same build toolchain as the current Windows 64-bit images >> (MSVC10) but located at different paths so you'll notice conditional >> checks like >> http://hg.mozilla.org/mozilla-central/file/a141e39bf6da/build/win64/mozconfig.vs2010#l2 >> which will be uplifted. >> >> The 'cedar' project branch has been building successfully using the new >> image. The plan is to migrate other branches over the next few weeks in >> this order: >> - remaining project branches >> - mozilla-central >> - mozilla-inbound, b2g-inbound >> - try >> - aurora >> - beta >> - release, mozilla-b2g18* >> - esr >> >> Wait times may increase for a short time on days that we cut over to the >> new build machines (since we are reimaging existing machines in chunks). >> We'll be keeping an eye on those wait times and adjusting the number of >> machines cut over if needed. Thunderbird comm-* branches will be cut >> over at the same time as their mozilla-* counterpart. >> >> The project branches will be cut over tomorrow and we'll let those build >> over the weekend before cutting over mozilla-central. I will send an >> email notification on this thread before and after a cutover so everyone >> knows the current status. >> >> Bug 781277 tracks this effort. If you have any questions or concerns, >> please let me know. >> >> Thanks, >> John >> > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Migrating to Win64 rev2 machines
Project branches are cutting over to the new win64-rev2 build machines. John On 13-10-09 5:22 PM, John Hopkins wrote: > Release Engineering is migrating the Windows 64-bit _build_ machines to > a new, managed machine image which will decrease the amount of > administrative overhead needed to deploy new instances, make image > changes, etc. > > It uses the same build toolchain as the current Windows 64-bit images > (MSVC10) but located at different paths so you'll notice conditional > checks like > http://hg.mozilla.org/mozilla-central/file/a141e39bf6da/build/win64/mozconfig.vs2010#l2 > which will be uplifted. > > The 'cedar' project branch has been building successfully using the new > image. The plan is to migrate other branches over the next few weeks in > this order: > - remaining project branches > - mozilla-central > - mozilla-inbound, b2g-inbound > - try > - aurora > - beta > - release, mozilla-b2g18* > - esr > > Wait times may increase for a short time on days that we cut over to the > new build machines (since we are reimaging existing machines in chunks). > We'll be keeping an eye on those wait times and adjusting the number of > machines cut over if needed. Thunderbird comm-* branches will be cut > over at the same time as their mozilla-* counterpart. > > The project branches will be cut over tomorrow and we'll let those build > over the weekend before cutting over mozilla-central. I will send an > email notification on this thread before and after a cutover so everyone > knows the current status. > > Bug 781277 tracks this effort. If you have any questions or concerns, > please let me know. > > Thanks, > John > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Migrating to Win64 rev2 machines
Hi, Jet: On 13-10-09 5:35 PM, Jet Villegas wrote: > I'm currently looking for someone to green up the tests on Win64. Can we > dedicate one machine instance (I'm assuming this is on AWS?) for the purpose > of advancing this greenery project? Also, who is the point-person on Release > Engineering that we should pull into the testing discussion? This switchover is specifically for the build machines, not test machines. You probably want bug 886640. Thanks, John > - Original Message ----- > From: "John Hopkins" > To: "dev.platform" > Cc: sheri...@mozilla.com > Sent: Wednesday, October 9, 2013 2:22:52 PM > Subject: Migrating to Win64 rev2 machines > > Release Engineering is migrating the Windows 64-bit _build_ machines to > a new, managed machine image which will decrease the amount of > administrative overhead needed to deploy new instances, make image > changes, etc. > > It uses the same build toolchain as the current Windows 64-bit images > (MSVC10) but located at different paths so you'll notice conditional > checks like > http://hg.mozilla.org/mozilla-central/file/a141e39bf6da/build/win64/mozconfig.vs2010#l2 > which will be uplifted. > > The 'cedar' project branch has been building successfully using the new > image. The plan is to migrate other branches over the next few weeks in > this order: > - remaining project branches > - mozilla-central > - mozilla-inbound, b2g-inbound > - try > - aurora > - beta > - release, mozilla-b2g18* > - esr > > Wait times may increase for a short time on days that we cut over to the > new build machines (since we are reimaging existing machines in chunks). > We'll be keeping an eye on those wait times and adjusting the number of > machines cut over if needed. Thunderbird comm-* branches will be cut > over at the same time as their mozilla-* counterpart. > > The project branches will be cut over tomorrow and we'll let those build > over the weekend before cutting over mozilla-central. I will send an > email notification on this thread before and after a cutover so everyone > knows the current status. > > Bug 781277 tracks this effort. If you have any questions or concerns, > please let me know. > > Thanks, > John > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Migrating to Win64 rev2 machines
Release Engineering is migrating the Windows 64-bit _build_ machines to a new, managed machine image which will decrease the amount of administrative overhead needed to deploy new instances, make image changes, etc. It uses the same build toolchain as the current Windows 64-bit images (MSVC10) but located at different paths so you'll notice conditional checks like http://hg.mozilla.org/mozilla-central/file/a141e39bf6da/build/win64/mozconfig.vs2010#l2 which will be uplifted. The 'cedar' project branch has been building successfully using the new image. The plan is to migrate other branches over the next few weeks in this order: - remaining project branches - mozilla-central - mozilla-inbound, b2g-inbound - try - aurora - beta - release, mozilla-b2g18* - esr Wait times may increase for a short time on days that we cut over to the new build machines (since we are reimaging existing machines in chunks). We'll be keeping an eye on those wait times and adjusting the number of machines cut over if needed. Thunderbird comm-* branches will be cut over at the same time as their mozilla-* counterpart. The project branches will be cut over tomorrow and we'll let those build over the weekend before cutting over mozilla-central. I will send an email notification on this thread before and after a cutover so everyone knows the current status. Bug 781277 tracks this effort. If you have any questions or concerns, please let me know. Thanks, John ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Migrating linux32 and linux64 builds to the cloud
In an earlier blog post[1] I explained how Release Engineering are moving the mozilla-central, try, and project branch Linux 32- and 64-bit builds to our new platform. This change allows us to do builds in the cloud as required and in response to build demand. Now that testing is complete, we are planning to deploy this change tomorrow (September 20). Thanks, John [1] http://johnnybuild.blogspot.ca/2012/08/migrating-linux32-and-linux64-builds-to.html ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
