Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
> I've edited the page, called Greg's proposal "Alternative A", and > added "Alternative B" which provides a little bit of time for catching > issues on mw.org: > > https://wikitech.wikimedia.org/wiki/Deployments/One_week Take a look at the page now, a few things you'll notice: 0) (very very) basic stats on our current cycle. 1) Each proposal now has a deployments calendar (the wikitable) showing which day of the week there are deploys, and what happens during those deploys. 2) There is now also a lifespan picture (sorry for the glare!) show, well, the lifespan of each wmfXX branch on production servers along with "development time" (ie: how many days between each wmfXX branch creation (ie: how many days of development per branch)). Take aways Option A * Devel time: 7 days * Lifespan: 10 days Option B * Devel time: 7 days * Lifespan: 14 days 3) There are now Pros/Cons tables for each option that outline what you think they outline. 4) Also, lastly, James_F kindly was the one to create the Talk: page with a clarification question on automated tests. That's the highlight of changes for that page so you can go look at it again and see the interesting bits (if any) for you. I'll reply separately with my recommendation. Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
On Mon, 2013-05-06 at 12:52 -0700, Greg Grossmeier wrote:
> Also, the majority of bugs that
> are in the Highest/Immediate priority level (from my gut assessment, I
> don't have the data here) are found after a deploy to non-WP projects.
I agree with that impression: We don't get many (manually found)
highest/immediate prio bug reports after the first deployment phase,
most of them after phase 2, and a few after phase 3 (e.g. when we failed
to understand the explosive force of an issue).
Backing that impression up with Bugzilla data:
: That's hard.
Long version:
I tried a Bugzilla query for tickets created in the last four months,
that at some point in their lifetime had Priority = {Highest |
Immediate}, restricted it to the products {MediaWiki, MediaWiki
extensions, Wikimedia}, made buglist.cgi display the "Opened" column
(via "Change columns" at the bottom); dropped the last 9 characters of
the "Opened" column (to get rid of the time and only have the date,
though that's UTC so does not perfectly fit our deployment *time*),
imported the resulting CSV into OOCalc, cumulated a bit, and summed up
all those tickets that got filed in a certain deployment phase && at
*some* point became highest/immediate. See attachment.
The results don't back up my impression.
One potential reason: Development teams file tickets *at some point* and
don't see priority immediately, and when tickets get triaged they get
higher priority at some point later on. Maybe results would look
different if I the query excluded reporters that are employees?
Don't want to spend too much time trying though.
andre
--
Andre Klapper | Wikimedia Bugwrangler
http://blogs.gnome.org/aklapper/
<>___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
> On Mon, May 6, 2013 at 3:52 PM, Greg Grossmeier wrote: > > I'm pretty sure this email will have a mid-air collision with Robla and > > his writeup of the proposal you outlined, Brad. > > I think Robla was the one who suggested it in February, actually. I > was just repeating it here. Right, didn't mean to suggest otherwise. Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
On Mon, May 6, 2013 at 3:52 PM, Greg Grossmeier wrote: > I'm pretty sure this email will have a mid-air collision with Robla and > his writeup of the proposal you outlined, Brad. I think Robla was the one who suggested it in February, actually. I was just repeating it here. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
On Mon, May 6, 2013 at 12:52 PM, Greg Grossmeier wrote: > Yep, Robla's creating that sample calendar now. > > My reasoning for removing that initial separation that may be faulty: > > We're running automated and non-automated tests against betalabs, and > that has started producing results (bugs reported and fixed before it > was ever included in a wmfXX branch). Also, the majority of bugs that > are in the Highest/Immediate priority level (from my gut assessment, I > don't have the data here) are found after a deploy to non-WP projects. > > Thus, in a way, the non-WP projects really are our betatesters in all > practicalities. That's my opinion at least. I'll work with Andre to get > some numbers on this, if we can (date blocker bugs reported against our > historic deploy calendar). > > I'm pretty sure this email will have a mid-air collision with Robla and > his writeup of the proposal you outlined, Brad. > I'm all for a move to a one-week cycle. I think it's a good sign that even discussing moving to a one-week cycle brings up a review of how and where we're catching critical bugs, ala the above comments from Greg, Sumana, etc. Weekly deployments pushing us all to be more diligent about this stuff sounds like a good thing, even if it may feel intense pre/post the switch in the short term. -- Steven Walling https://wikimediafoundation.org/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
> > One other plan that was discussed in February, if I recall, was > something like this: > > week0, Thursday: Deploy wmf1 to test and mw.org > week1, Monday: Deploy wmf1 to first-round wikis > week1, Thursday: Deploy wmf1 to remaining wikis and wmf2 to test and mw.org. > week2, Monday: Deploy wmf2 to first-round wikis > week2, Thursday: Deploy wmf2 to remaining wikis and wmf3 to test and mw.org. > etc. > > That has the advantage of preserving the separation of test+mw.org > from the more user-focused wikis, as Sumana mentioned. Yep, Robla's creating that sample calendar now. My reasoning for removing that initial separation that may be faulty: We're running automated and non-automated tests against betalabs, and that has started producing results (bugs reported and fixed before it was ever included in a wmfXX branch). Also, the majority of bugs that are in the Highest/Immediate priority level (from my gut assessment, I don't have the data here) are found after a deploy to non-WP projects. Thus, in a way, the non-WP projects really are our betatesters in all practicalities. That's my opinion at least. I'll work with Andre to get some numbers on this, if we can (date blocker bugs reported against our historic deploy calendar). I'm pretty sure this email will have a mid-air collision with Robla and his writeup of the proposal you outlined, Brad. Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
On Mon, May 6, 2013 at 12:20 PM, Sumana Harihareswara wrote: > Extrapolating from our experience from two-week deploy cycles and what > bugs we find at each stage, would moving to a one-week deploy cycle > substantially increase the number of users exposed to really bad bugs? > > For instance, if we're currently finding and fixing about 2 deployment > blocker bugs per cycle after the mediawiki.org deploy but before the > non-Wikipedias deploy, that data might imply that we should structure a > 1-week cycle differently -- perhaps with three stages rather than two. I don't know what the stats are, but my intuition is that it's not generally that high. However, the ones we find are kinda big, so it still makes me nervous to go that route. I've edited the page, called Greg's proposal "Alternative A", and added "Alternative B" which provides a little bit of time for catching issues on mw.org: https://wikitech.wikimedia.org/wiki/Deployments/One_week Rob ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
On Mon, May 6, 2013 at 3:04 PM, Greg Grossmeier wrote: > > Let's move our MediaWiki deploy cycle to weekly instead of 2-week. > This will also reduce the number of standing deployment windows > throughout the week by having those projects/teams simply "ride the > MediaWiki train." \o/ > It takes up to 2 weeks for new features/bug fixes to be rolled out to > the various Wikimedia wikis. 3.5 weeks, actually. Consider if something was merged in the afternoon on April 29. It just missed getting into 1.22wmf3, so absent backporting it would have to wait for 1.22wmf4, which finishes being deployed everywhere on May 22. > This has been talked about a bit, including during the last In Town Week > for WMF Engineering in late-February. I've coalesced on one proposal at: > https://wikitech.wikimedia.org/wiki/Deployments/One_week > > This seems like a reasonable approach to me. Please respond here or on > the talk page with comments/suggestions/etc. One other plan that was discussed in February, if I recall, was something like this: week0, Thursday: Deploy wmf1 to test and mw.org week1, Monday: Deploy wmf1 to first-round wikis week1, Thursday: Deploy wmf1 to remaining wikis and wmf2 to test and mw.org. week2, Monday: Deploy wmf2 to first-round wikis week2, Thursday: Deploy wmf2 to remaining wikis and wmf3 to test and mw.org. etc. That has the advantage of preserving the separation of test+mw.org from the more user-focused wikis, as Sumana mentioned. -- Brad Jorsch Software Engineer Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Engineering] Proposal: Let's move to a one-week deploy cycle
Extrapolating from our experience from two-week deploy cycles and what bugs we find at each stage, would moving to a one-week deploy cycle substantially increase the number of users exposed to really bad bugs? For instance, if we're currently finding and fixing about 2 deployment blocker bugs per cycle after the mediawiki.org deploy but before the non-Wikipedias deploy, that data might imply that we should structure a 1-week cycle differently -- perhaps with three stages rather than two. -- Sumana Harihareswara Engineering Community Manager Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
